Oct 30 03:00:03.108401 ip-10-0-135-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 03:00:03.108412 ip-10-0-135-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 03:00:03.108419 ip-10-0-135-226 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 03:00:03.108648 ip-10-0-135-226 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 03:00:13.350589 ip-10-0-135-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 03:00:13.350607 ip-10-0-135-226 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ef77d2a0d2bf434d9df5e7d8324b997f --
Oct 30 03:02:10.528691 ip-10-0-135-226 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 03:02:10.925369 ip-10-0-135-226 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:02:10.925369 ip-10-0-135-226 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 03:02:10.925369 ip-10-0-135-226 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:02:10.925369 ip-10-0-135-226 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:02:10.930560 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930481    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.930560 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930560    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930564    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930567    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930570    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930573    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930575    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930578    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930581    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930583    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930586    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930589    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930591    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930594    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930597    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930600    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930602    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930605    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930607    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930610    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930613    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.930629 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930615    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930618    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930620    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930623    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930626    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930628    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930631    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930639    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930643    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930645    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930648    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930651    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930654    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930657    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930660    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930663    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930666    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930668    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930671    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930674    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.931081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930676    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930678    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930681    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930683    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930686    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930688    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930691    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930693    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930696    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930699    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930701    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930704    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930706    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930709    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930712    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930714    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930717    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930720    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930723    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.931580 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930726    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930728    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930731    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930733    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930737    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930740    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930743    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930751    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930754    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930756    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930759    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930762    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930764    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930767    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930769    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930773    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930777    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930781    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930784    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.932034 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930787    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930791    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930795    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930798    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930801    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930804    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930806    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930809    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930813    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930815    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930818    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930821    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930823    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930826    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930829    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930831    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930834    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930838    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930841    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930843    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.932527 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930846    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930848    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930851    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930853    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930856    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930858    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.930861    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931668    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931677    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931680    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931683    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931687    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931689    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931692    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931695    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931698    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931701    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931703    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931706    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931709    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.933002 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931712    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931715    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931718    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931722    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931726    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931729    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931734    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931737    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931740    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931743    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931746    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931750    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931753    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931756    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931759    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931762    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931764    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931767    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931769    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.933489 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931772    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931775    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931777    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931780    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931782    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931785    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931787    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931790    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931801    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931804    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931807    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931810    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931812    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931815    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931818    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931820    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931824    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931827    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931830    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.933951 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931832    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931835    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931838    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931840    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931843    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931845    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931848    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931850    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931853    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931856    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931859    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931861    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931864    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931866    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931869    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931871    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931874    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931876    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931879    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931881    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.934408 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931884    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931886    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931889    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931892    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931895    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931897    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931900    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931904    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931906    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931909    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931912    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931915    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931917    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931920    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931922    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931925    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931928    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931930    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931932    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931935    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.934895 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931938    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931940    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931943    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931945    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931948    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931950    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931953    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931955    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931958    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931960    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931963    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931965    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931968    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931970    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.931973    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932451    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932458    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932464    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932467    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932469    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.935372 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932472    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932475    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932478    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932481    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932483    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932486    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932488    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932491    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932493    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932496    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932499    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932501    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932504    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932507    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932509    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932512    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932514    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932517    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932519    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932522    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.935856 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932524    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932527    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932530    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932532    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932535    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932538    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932540    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932544    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932546    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932550    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932553    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932555    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932558    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932560    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932563    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932566    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932569    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932571    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932575    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.936342 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932579    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932582    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932585    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932588    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932590    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932593    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932595    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932598    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932600    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932603    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932606    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932608    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932612    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932617    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932620    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932623    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932625    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932628    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932631    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.936823 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932633    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932636    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932639    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932643    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932646    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932648    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932651    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932653    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932656    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932658    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932661    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932664    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932666    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932669    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932671    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932674    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932677    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932679    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932682    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932685    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.937259 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932688    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932690    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932693    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932695    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932697    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932700    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932703    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932705    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932708    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932710    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932712    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932715    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932717    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932720    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932722    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932727    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932729    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932732    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932734    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.937803 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932737    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932739    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932743    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.932745    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933855    2586 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933864    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933870    2586 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933875    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933880    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933886    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933890    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933894    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933897    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933900    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933904    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933907    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933911    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933913    2586 flags.go:64] FLAG: --cgroup-root=""
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933916    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933920    2586 flags.go:64] FLAG: --client-ca-file=""
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933922    2586 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933925    2586 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933931    2586 flags.go:64] FLAG: --cluster-domain=""
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933935    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 03:02:10.938254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933938    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933941    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933951    2586 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933956    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933959    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933964    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933967    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933971    2586 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933974    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933977    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933980    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933983    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933987    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933990    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933993    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933996    2586 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.933999    2586 flags.go:64] FLAG: --enable-server="true"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934004    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934007    2586 flags.go:64] FLAG: --event-burst="100"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934011    2586 flags.go:64] FLAG: --event-qps="50"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934014    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934017    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934020    2586 flags.go:64] FLAG: --eviction-hard=""
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934024    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 03:02:10.938931 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934027    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934030    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934033    2586 flags.go:64] FLAG: --eviction-soft=""
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934036    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934039    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934042    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934045    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934048    2586 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934051    2586 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934054    2586 flags.go:64] FLAG: --feature-gates=""
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934058    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934061    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934065    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934068    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934072    2586 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934075    2586 flags.go:64] FLAG: --help="false"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934078    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934082    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934085    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934088    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934091    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934094    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934098    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 03:02:10.939589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934101    2586 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934103    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934108    2586 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934111    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934114    2586 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934117    2586 flags.go:64] FLAG: --kube-reserved=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934120    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934123    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934126    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934130    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934133    2586 flags.go:64] FLAG: --lock-file=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934135    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934138    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934141    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934147    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934150    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934153    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934156    2586 flags.go:64] FLAG: --logging-format="text"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934159    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934162    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934165    2586 flags.go:64] FLAG: --manifest-url=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934168    2586 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934173    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934176    2586 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934181    2586 flags.go:64] FLAG: --max-pods="110"
Oct 30 03:02:10.940155 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934184    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934187    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934190    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934192    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934195    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934198    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934201    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934208    2586 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934212    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934217    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934220    2586 flags.go:64] FLAG: --pod-cidr=""
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934223    2586 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934226    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934230    2586 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934233    2586 flags.go:64] FLAG: --port="10250"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934236    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934239    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027b9566ec751b6e2"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934242    2586 flags.go:64] FLAG: --qos-reserved=""
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934245    2586 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934248    2586 flags.go:64] FLAG: --register-node="true"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934251    2586 flags.go:64] FLAG: --register-with-taints=""
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934255    2586 flags.go:64] FLAG: --registry-burst="10"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934258    2586 flags.go:64] FLAG: --registry-qps="5"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934261    2586 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934263    2586 flags.go:64] FLAG: --reserved-memory=""
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934267    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 03:02:10.940763 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934270    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934274    2586 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934277    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934279    2586 flags.go:64] FLAG: --runonce="false"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934282    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934286    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934289    2586 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934292    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934295    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934298    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934301    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934304    2586 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934308    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934310    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934313    2586 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934317    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934321    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934324    2586 flags.go:64] FLAG: --system-cgroups=""
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934326    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934332    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934335    2586 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934338    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934342    2586 flags.go:64] FLAG: --tls-min-version=""
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934345    2586 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934348    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 03:02:10.941392 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934351    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934354    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934357    2586 flags.go:64] FLAG: --v="2"
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934361    2586 flags.go:64] FLAG: --version="false"
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934365    2586 flags.go:64] FLAG: --vmodule=""
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934369    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934372    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934510    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934516    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934519    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934522    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934525    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934528    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934530    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934534    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934536    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934539    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934542    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934545    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934548    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934550    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.942012 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934553    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934557    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934560    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934563    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934566    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934568    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934571    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934574    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934576    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934579    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934581    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934584    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934587    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934589    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934592    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934594    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934597    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934600    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934602    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934606    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.942727 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934609    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934611    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934614    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934616    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934619    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934621    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934624    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934627    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934629    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934632    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934634    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934637    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934639    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934643    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934646    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934648    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934651    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934655    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934659    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.943569 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934662    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934666    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934669    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934672    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934674    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934677    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934679    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934682    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934685    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934687    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934690    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934692    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934697    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934699    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934702    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934705    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934707    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934710    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934712    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.944368 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934715    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934718    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934720    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934723    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934725    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934728    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934730    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934734    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934737    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934739    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934742    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934744    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934747    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934749    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934753    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934757    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934760    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934762    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934765    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934768    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.945035 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934771    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934774    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934776    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934779    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934782    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934785    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934788    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934790    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934793    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934795    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934798    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934800    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934803    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934806    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.934831    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934870    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.945655 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934875    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934878    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934882    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934885    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934889    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934892    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934895    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934899    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934902    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934904    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934907    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934910    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934913    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934915    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934918    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934920    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934923    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934926    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934928    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934931    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.946337 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934933    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934937    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934939    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934942    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934945    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934947    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934950    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934952    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934955    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934958    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934960    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934964    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934967    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934970    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934972    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934975    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934978    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934981    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934983    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.947218 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934986    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934988    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934991    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934993    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934996    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.934998    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935001    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935005    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935008    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935012    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935015    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935018    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935020    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935023    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935027    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935029    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935032    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935034    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935037    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.948774 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935039    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935042    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935044    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935047    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935049    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935052    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935055    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935057    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935060    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935062    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935065    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935068    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935070    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935073    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935075    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935078    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935080    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935083    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935085    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.949557 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935088    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935090    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935093    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935096    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935100    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935103    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935106    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935109    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935113    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935116    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935118    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935121    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935123    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935126    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935129    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935131    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935134    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935136    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935139    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.950128 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935141    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935144    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935146    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935150    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935153    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935155    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935158    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935160    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.935163    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.944734    2586 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.944755    2586 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.945537    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.945551    2586 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945637    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945645    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945650    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945654    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945659    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945663    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945667    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.950714 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945672    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945676    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945681    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945685    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945690    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945694    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945698    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945703    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945707    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945712    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945716    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945720    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945724    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945728    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945733    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945737    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945742    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945746    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945750    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.951216 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945757    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945764    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945769    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945777    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945782    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945787    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945791    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945795    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945800    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945804    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945808    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945813    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945817    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945822    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945826    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945830    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945835    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945839    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945843    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.951769 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945848    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945852    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945857    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945861    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945866    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945870    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945874    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945879    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945883    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945887    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945892    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945896    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945900    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945904    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945908    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945912    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945917    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945923    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945927    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945932    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.952237 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945936    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945941    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945953    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945957    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945961    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945966    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945970    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945974    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945978    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945984    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945990    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945995    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.945999    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946004    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946008    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946012    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946016    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946020    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946024    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.952874 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946029    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946033    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946038    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946042    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946047    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946051    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946055    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946059    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946063    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946069    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946073    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946078    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946082    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946087    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946091    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946096    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946100    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946104    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946109    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946113    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.953346 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946118    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946122    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.946165    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946384    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946392    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946397    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946401    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946405    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946410    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946414    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946418    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946440    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946444    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946449    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946453    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946457    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.953990 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946461    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946466    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946470    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946475    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946480    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946484    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946489    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946494    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946499    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946504    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946509    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946516    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946521    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946526    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946530    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946535    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946539    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946543    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946547    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.954664 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946551    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946556    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946560    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946565    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946569    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946574    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946578    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946582    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946586    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946590    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946594    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946599    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946604    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946608    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946612    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946616    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946621    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946625    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946629    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.955289 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946633    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946640    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946646    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946653    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946658    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946662    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946667    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946671    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946675    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946679    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946683    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946688    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946692    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946697    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946701    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946705    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946709    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946714    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946718    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.955804 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946722    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946726    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946731    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946736    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946740    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946744    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946749    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946753    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946757    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946762    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946766    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946771    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946775    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946780    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946784    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946788    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946792    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946799    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946803    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946807    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.956246 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946812    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946816    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946820    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946824    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946828    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946833    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946837    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946841    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946845    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946850    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946854    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946859    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946863    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946867    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946871    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.946876    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.956834 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.946915    2586 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947001    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947010    2586 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947015    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947020    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947025    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947030    2586 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947034    2586 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947038    2586 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947043    2586 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947047    2586 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947051    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947055    2586 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947060    2586 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947065    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947070    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947075    2586 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947079    2586 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947083    2586 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:10.957211 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947088    2586 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947092    2586 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947098    2586 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947104    2586 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947109    2586 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947113    2586 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947117    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947122    2586 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947126    2586 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947130    2586 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947135    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947139    2586 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947143    2586 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947147    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947152    2586 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947156    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947160    2586 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947165    2586 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947169    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:10.957694 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947173    2586 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947178    2586 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947182    2586 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947186    2586 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947191    2586 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947195    2586 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947199    2586 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947203    2586 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947208    2586 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947216    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947221    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947225    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947229    2586 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947233    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947237    2586 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947241    2586 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947245    2586 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947250    2586 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947256    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947260    2586 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:10.958149 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947265    2586 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947269    2586 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947273    2586 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947277    2586 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947281    2586 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947285    2586 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947289    2586 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947294    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947298    2586 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947303    2586 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947307    2586 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947311    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947316    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947320    2586 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947324    2586 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947328    2586 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947333    2586 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947337    2586 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947342    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:10.958622 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947346    2586 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947351    2586 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947357    2586 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947362    2586 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947366    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947370    2586 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947374    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947379    2586 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947383    2586 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947387    2586 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947392    2586 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947396    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947400    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947404    2586 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947408    2586 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947412    2586 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947417    2586 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947437    2586 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947441    2586 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947446    2586 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:10.959106 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947450    2586 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947455    2586 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947463    2586 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947468    2586 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947473    2586 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947480    2586 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947484    2586 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947489    2586 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947493    2586 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:10.947498    2586 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.947574    2586 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.950585    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.951724    2586 server.go:1014] "Starting client certificate rotation"
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.951825    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.951860    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.955760    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:02:10.959577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.958980    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:02:10.969972 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.969957    2586 log.go:25] "Validated CRI v1 runtime API"
Oct 30 03:02:10.974902 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.974877    2586 log.go:25] "Validated CRI v1 image API"
Oct 30 03:02:10.975761 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.975740    2586 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 03:02:10.979133 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.979114    2586 fs.go:135] Filesystem UUIDs: map[2c131d26-2f91-4c0c-aa60-f90299cd035d:/dev/nvme0n1p4 6c3df54c-3031-467d-b89f-5cc08c247aeb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 03:02:10.979224 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.979133    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 03:02:10.983314 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.983195    2586 manager.go:217] Machine: {Timestamp:2026-10-30 03:02:10.982745583 +0000 UTC m=+0.366589047 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100451 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e38677ed165f104fe4b7b80cd85eb SystemUUID:ec2e3867-7ed1-65f1-04fe-4b7b80cd85eb BootID:ef77d2a0-d2bf-434d-9df5-e7d8324b997f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:9f:70:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:9f:70:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:8c:dd:99:7b:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 03:02:10.983314 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.983304    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 03:02:10.983490 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.983394    2586 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 03:02:10.987163 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.987142    2586 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 03:02:10.987283 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.987165    2586 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 03:02:10.987327 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.987285    2586 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 03:02:10.987327 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.987293    2586 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 03:02:10.987327 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.987309    2586 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:02:10.987327 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.987326    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:02:10.988618 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.988608    2586 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 03:02:10.993394 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.993382    2586 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 03:02:10.993528 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.993519    2586 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 03:02:10.994364 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.994355    2586 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 03:02:10.994396 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.994368    2586 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 03:02:10.994396 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.994380    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 03:02:10.994396 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.994389    2586 kubelet.go:405] "Adding apiserver pod source"
Oct 30 03:02:10.994506 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.994398    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 03:02:10.994736 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.994721    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:02:10.997276 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:10.997260    2586 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 03:02:11.004623 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.004606    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:02:11.005544 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005528    2586 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 03:02:11.005600 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005567    2586 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 03:02:11.005769 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005758    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 03:02:11.005823 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005778    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 03:02:11.005823 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005789    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 03:02:11.005823 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005797    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 03:02:11.005823 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005806    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 03:02:11.005823 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.005815    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 03:02:11.006532 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.006519    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 03:02:11.006562 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.006537    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 03:02:11.006562 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.006549    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 03:02:11.006562 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.006557    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 03:02:11.006638 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.006572    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 03:02:11.006638 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.006585    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 03:02:11.007520 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.007508    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 03:02:11.007555 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.007523    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 03:02:11.011213 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.011200    2586 server.go:1263] "Started kubelet"
Oct 30 03:02:11.011340 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.011317    2586 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 03:02:11.011375 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.011302    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 03:02:11.011375 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.011369    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 03:02:11.011836 ip-10-0-135-226 systemd[1]: Started Kubernetes Kubelet.
Oct 30 03:02:11.012860 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.012840    2586 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 03:02:11.013319 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.013304    2586 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 03:02:11.019727 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.019704    2586 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 03:02:11.022143 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.022123    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 03:02:11.022796 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.022783    2586 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 03:02:11.022865 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.022848    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.023537 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023519    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 03:02:11.023604 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023516    2586 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 03:02:11.023604 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023559    2586 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 03:02:11.023697 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023673    2586 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 03:02:11.023697 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023681    2586 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 03:02:11.023860 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023843    2586 factory.go:55] Registering systemd factory
Oct 30 03:02:11.023860 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.023862    2586 factory.go:223] Registration of the systemd container factory successfully
Oct 30 03:02:11.024097 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.024084    2586 factory.go:153] Registering CRI-O factory
Oct 30 03:02:11.024168 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.024101    2586 factory.go:223] Registration of the crio container factory successfully
Oct 30 03:02:11.024168 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.024154    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 03:02:11.024261 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.024175    2586 factory.go:103] Registering Raw factory
Oct 30 03:02:11.024261 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.024190    2586 manager.go:1199] Started watching for new ooms in manager
Oct 30 03:02:11.025328 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.025312    2586 manager.go:319] Starting recovery of all containers
Oct 30 03:02:11.032400 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.032369    2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 03:02:11.036695 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.036679    2586 manager.go:324] Recovery completed
Oct 30 03:02:11.042171 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.042158    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.044614 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.044597    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.044694 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.044626    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.044694 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.044636    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.045174 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.045161    2586 cpu_manager.go:226] "Starting" policy="none"
Oct 30 03:02:11.045174 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.045171    2586 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 03:02:11.045254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.045178    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.045254 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.045192    2586 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 03:02:11.045899 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.045874    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 03:02:11.046286 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.046240    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.046286 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.046279    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.046382 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.046295    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.047069 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.045881    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-226.ec2.internal.18e32f866c35c047  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-226.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-226.ec2.internal,},FirstTimestamp:2026-10-30 03:02:11.011174471 +0000 UTC m=+0.395017935,LastTimestamp:2026-10-30 03:02:11.011174471 +0000 UTC m=+0.395017935,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-226.ec2.internal,}"
Oct 30 03:02:11.047367 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.047354    2586 policy_none.go:50] "Start"
Oct 30 03:02:11.047465 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.047370    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 03:02:11.047465 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.047387    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 03:02:11.057674 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.057655    2586 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 03:02:11.062559 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.062478    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-226.ec2.internal.18e32f866e33fe04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-226.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-226.ec2.internal,},FirstTimestamp:2026-10-30 03:02:11.044613636 +0000 UTC m=+0.428457104,LastTimestamp:2026-10-30 03:02:11.044613636 +0000 UTC m=+0.428457104,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-226.ec2.internal,}"
Oct 30 03:02:11.062735 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.062584    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 03:02:11.062735 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.062598    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 03:02:11.062873 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.062591    2586 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 03:02:11.072253 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.072236    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pt89h"
Oct 30 03:02:11.080050 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.080029    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pt89h"
Oct 30 03:02:11.089981 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.089896    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-226.ec2.internal.18e32f866e344285  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-226.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-226.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-226.ec2.internal,},FirstTimestamp:2026-10-30 03:02:11.044631173 +0000 UTC m=+0.428474637,LastTimestamp:2026-10-30 03:02:11.044631173 +0000 UTC m=+0.428474637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-226.ec2.internal,}"
Oct 30 03:02:11.123858 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.123838    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.180975 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.180931    2586 policy_none.go:44] "Start"
Oct 30 03:02:11.217021 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.216993    2586 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 03:02:11.219338 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219324    2586 manager.go:341] "Starting Device Plugin manager"
Oct 30 03:02:11.219417 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.219405    2586 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 03:02:11.219417 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219419    2586 server.go:87] "Starting device plugin registration server"
Oct 30 03:02:11.219681 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219667    2586 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 03:02:11.219744 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219682    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 03:02:11.219808 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219795    2586 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 03:02:11.219864 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219855    2586 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 03:02:11.219864 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.219864    2586 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 03:02:11.220450 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.220417    2586 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 03:02:11.220527 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.220476    2586 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.238227 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.238205    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 03:02:11.239410 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.239394    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 03:02:11.239481 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.239435    2586 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 03:02:11.239481 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.239461    2586 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 03:02:11.239551 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.239501    2586 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 03:02:11.256895 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.256880    2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:11.293574 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.293555    2586 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-226.ec2.internal\" not found" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.320674 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.320657    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.321477 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.321463    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.321535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.321490    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.321535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.321500    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.321535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.321523    2586 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.340324 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.340302    2586 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"]
Oct 30 03:02:11.340405 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.340352    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.341164 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.341147    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.341217 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.341176    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.341217 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.341186    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.341591 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.341572    2586 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.341627 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.341599    2586 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-226.ec2.internal\": node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.343367 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.343356    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.343938 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.343922    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.343989 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.343958    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.344022 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.343995    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.344022 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.344018    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.344119 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.344032    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.344606 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.344588    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.344681 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.344616    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.344681 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.344629    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.346139 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.346124    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.346218 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.346152    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:11.346783 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.346768    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:11.346855 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.346798    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:11.346855 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.346810    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:11.378994 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.378978    2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-226.ec2.internal\" not found" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.381533 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.381515    2586 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-226.ec2.internal\" not found" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.384083 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.384070    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.425212 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.425193    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.485068 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.485024    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.525757 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.525738    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.525757 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.525752    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.525861 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.525771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.525861 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.525787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c7f2093544cf1327cc6ce2b1dd9d362-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2c7f2093544cf1327cc6ce2b1dd9d362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.585349 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.585324    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.626130 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.626111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.626186 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.626133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c7f2093544cf1327cc6ce2b1dd9d362-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2c7f2093544cf1327cc6ce2b1dd9d362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.626186 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.626180    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c7f2093544cf1327cc6ce2b1dd9d362-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2c7f2093544cf1327cc6ce2b1dd9d362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.626252 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.626199    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.682119 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.682084    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.684232 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.684212    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:11.685775 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.685756    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.786417 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.786214    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.886724 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.886700    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:11.951479 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.951458    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 03:02:11.951817 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:11.951590    2586 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 03:02:11.987384 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:11.987359    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:12.080924 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.080859    2586 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:12.086394 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.086369    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:57:11 +0000 UTC" deadline="2028-07-15 08:57:48.506245306 +0000 UTC"
Oct 30 03:02:12.086452 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.086394    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14981h55m36.419854511s"
Oct 30 03:02:12.087544 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:12.087529    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:12.188573 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:12.188546    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:12.190722 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.190707    2586 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:12.192550 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.192530    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:12.192632 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.192572    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:12.192632 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.192588    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:12.263706 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.263666    2586 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:12.288724 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:12.288689    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:12.371081 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:12.371050    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be6db401bef7646097e7da845f451bd.slice/crio-b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b WatchSource:0}: Error finding container b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b: Status 404 returned error can't find the container with id b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b
Oct 30 03:02:12.371581 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:12.371559    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7f2093544cf1327cc6ce2b1dd9d362.slice/crio-65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42 WatchSource:0}: Error finding container 65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42: Status 404 returned error can't find the container with id 65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42
Oct 30 03:02:12.376197 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.376181    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:02:12.389896 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:12.389868    2586 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:12.449204 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.449183    2586 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:12.523487 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.523462    2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:12.544166 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.544151    2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:02:12.544920 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.544907    2586 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:12.563605 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.563590    2586 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:02:12.995236 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:12.995192    2586 apiserver.go:52] "Watching apiserver"
Oct 30 03:02:13.022264 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.022236    2586 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 03:02:13.022589 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.022568    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ljjrp","openshift-dns/node-resolver-2hl5n","openshift-multus/network-metrics-daemon-z74kk","openshift-network-operator/iptables-alerter-2qcvz","openshift-image-registry/node-ca-z5sbs","openshift-multus/multus-additional-cni-plugins-8wqsd","openshift-multus/multus-czcg4","kube-system/global-pull-secret-syncer-rtd54","openshift-network-diagnostics/network-check-target-6ssh4","openshift-ovn-kubernetes/ovnkube-node-z54vn","kube-system/konnectivity-agent-sfv5t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"]
Oct 30 03:02:13.024056 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.024029    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 03:02:13.026037 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.026008    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.028550 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.028528    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.030841 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.030819    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.033344 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.033325    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.034148 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.034122    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da977d0-2885-4889-a18f-9038e7e11272-host\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.034255 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.034161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9da977d0-2885-4889-a18f-9038e7e11272-serviceca\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.034255 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.034186    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtjx\" (UniqueName: \"kubernetes.io/projected/9da977d0-2885-4889-a18f-9038e7e11272-kube-api-access-8xtjx\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.036350 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.036330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.036447 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.036406    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:13.038943 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.038623    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.041859 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.041838    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.044675 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.044656    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.047277 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.047254    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.049401 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.049381    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.049516 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.049469    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:13.051953 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.051923    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:13.052068 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.051975    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:13.054792 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.054772    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.056301 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056276    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.056413 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056393    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 03:02:13.056484 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056425    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.056484 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056455    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bl9p4\""
Oct 30 03:02:13.056735 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056715    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6fpwt\""
Oct 30 03:02:13.056810 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056732    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.056810 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.056740    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 03:02:13.063476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063443    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.063569 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063517    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.063635 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063595    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 03:02:13.063690 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063651    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc5sd\""
Oct 30 03:02:13.063752 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063692    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.063752 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063738    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 03:02:13.063860 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063792    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.063913 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063859    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.063966 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063914    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hhqhj\""
Oct 30 03:02:13.063966 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.063956    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 03:02:13.064078 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064012    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 03:02:13.064078 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064053    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.064179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064100    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j94bv\""
Oct 30 03:02:13.064179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064163    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7q2t\""
Oct 30 03:02:13.064274 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064250    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.064790 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064414    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mpbq7\""
Oct 30 03:02:13.064900 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064575    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 03:02:13.064900 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064605    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.065024 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064672    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 03:02:13.065024 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.064744    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.065535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.065515    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 03:02:13.065631 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.065568    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nc7qz\""
Oct 30 03:02:13.075958 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.075938    2586 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 03:02:13.076032 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.076018    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hbxxd"
Oct 30 03:02:13.085758 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.084793    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hbxxd"
Oct 30 03:02:13.124478 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.124461    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 03:02:13.124602 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.124578    2586 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:13.134830 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.134806    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-tuned\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.134941 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.134843    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-netns\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.134998 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.134964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9de66365-30cc-4adc-ad1b-4a6778246af8-hosts-file\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.135060 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.134998    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-kubelet\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.135060 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135026    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysconfig\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.135060 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135050    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a767b5d-c7e1-4178-8793-4bd9f4352745-iptables-alerter-script\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.135208 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135076    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.135208 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-slash\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.135208 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-bin\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.135208 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135154    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-device-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.135208 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135179    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-tmp\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.135208 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4bs\" (UniqueName: \"kubernetes.io/projected/8046eaad-8e51-49d2-beac-8f6be5170f4d-kube-api-access-sh4bs\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135271    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a767b5d-c7e1-4178-8793-4bd9f4352745-host-slash\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135336    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-dbus\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135373    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135401    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4lv\" (UniqueName: \"kubernetes.io/projected/63498c86-702e-47d4-8002-0693eb34b05e-kube-api-access-rv4lv\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135471    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9da977d0-2885-4889-a18f-9038e7e11272-serviceca\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135501    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-sys\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135527    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-var-lib-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.135558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135554    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-node-log\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135577    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-config\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135612    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-socket-dir-parent\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135658    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-bin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135696    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135725    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-sys-fs\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135758    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.135838 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135797    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-systemd-units\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135831    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-etc-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135866    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-ovn\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135913    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9da977d0-2885-4889-a18f-9038e7e11272-serviceca\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135909    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-socket-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135953    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-registration-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.135981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-system-cni-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136013    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.136049 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136045    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rvh\" (UniqueName: \"kubernetes.io/projected/060b1735-396e-43d2-940d-e5b9bd3ec06b-kube-api-access-49rvh\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136071    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-log-socket\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136121    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63498c86-702e-47d4-8002-0693eb34b05e-ovn-node-metrics-cert\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136137    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136189    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/940bd241-206c-44c1-9c91-b6eb875bce7c-konnectivity-ca\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136224    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-netns\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136251    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-host\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136267    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtjx\" (UniqueName: \"kubernetes.io/projected/9da977d0-2885-4889-a18f-9038e7e11272-kube-api-access-8xtjx\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136310    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136328    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/940bd241-206c-44c1-9c91-b6eb875bce7c-agent-certs\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-etc-kubernetes\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136386    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da977d0-2885-4889-a18f-9038e7e11272-host\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136406    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136423    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-var-lib-kubelet\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.136442 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-system-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136457    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkx9\" (UniqueName: \"kubernetes.io/projected/fdce563b-9436-449d-8cd6-159e7da40143-kube-api-access-zjkx9\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136477    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-env-overrides\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-script-lib\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136604    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-cni-binary-copy\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136621    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-k8s-cni-cncf-io\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136642    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-multus-daemon-config\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136663    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136679    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-kubernetes\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136696    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-lib-modules\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136711    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cj5d\" (UniqueName: \"kubernetes.io/projected/9de66365-30cc-4adc-ad1b-4a6778246af8-kube-api-access-5cj5d\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136726    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-kubelet-config\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-conf-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136761    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de66365-30cc-4adc-ad1b-4a6778246af8-tmp-dir\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136778    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136808    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-hostroot\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136823    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-multus-certs\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.137250 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136892    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-conf\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136909    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-kubelet\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136928    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-cnibin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136949    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-multus\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136966    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd8sr\" (UniqueName: \"kubernetes.io/projected/e00a6bb6-263b-40c7-977d-11511832d40a-kube-api-access-jd8sr\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.136999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-cnibin\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137023    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-modprobe-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-systemd\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137053    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vqp\" (UniqueName: \"kubernetes.io/projected/9a767b5d-c7e1-4178-8793-4bd9f4352745-kube-api-access-x2vqp\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137071    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-systemd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-netd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137111    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137133    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjfw\" (UniqueName: \"kubernetes.io/projected/66f7168c-a3bc-43e5-a198-b3551b33b770-kube-api-access-fjjfw\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137150    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-run\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.138089 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.137396    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da977d0-2885-4889-a18f-9038e7e11272-host\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.153100 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.153074    2586 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 03:02:13.157238 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.157216    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtjx\" (UniqueName: \"kubernetes.io/projected/9da977d0-2885-4889-a18f-9038e7e11272-kube-api-access-8xtjx\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.164304 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164283    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 03:02:13.164409 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164330    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 03:02:13.164409 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164386    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 03:02:13.164521 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164454    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 03:02:13.164521 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164473    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 03:02:13.164744 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164729    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f9hw2\""
Oct 30 03:02:13.164880 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.164861    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 03:02:13.237874 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.237839    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-host\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.238018 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.237894    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.238018 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.237925    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.238018 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.237952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/940bd241-206c-44c1-9c91-b6eb875bce7c-agent-certs\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.238018 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.237979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-etc-kubernetes\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238018 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238039    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-var-lib-kubelet\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238067    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-system-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkx9\" (UniqueName: \"kubernetes.io/projected/fdce563b-9436-449d-8cd6-159e7da40143-kube-api-access-zjkx9\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238122    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-env-overrides\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-script-lib\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238181    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-cni-binary-copy\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-k8s-cni-cncf-io\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238216    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-multus-daemon-config\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238232    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-kubernetes\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238264    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-lib-modules\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238281    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cj5d\" (UniqueName: \"kubernetes.io/projected/9de66365-30cc-4adc-ad1b-4a6778246af8-kube-api-access-5cj5d\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-kubelet-config\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238312    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-conf-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238336    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de66365-30cc-4adc-ad1b-4a6778246af8-tmp-dir\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-hostroot\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238389    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-multus-certs\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238404    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-conf\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-kubelet\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238434    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-cnibin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238449    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-multus\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238464    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8sr\" (UniqueName: \"kubernetes.io/projected/e00a6bb6-263b-40c7-977d-11511832d40a-kube-api-access-jd8sr\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238508    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.238595 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238535    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-cnibin\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238563    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-modprobe-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238582    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238610    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-conf-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238626    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-var-lib-kubelet\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238669    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-systemd\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238683    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-modprobe-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238682    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-kubelet-config\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238724    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238790    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-kubernetes\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238627    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-systemd\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238830    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-conf\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238854    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vqp\" (UniqueName: \"kubernetes.io/projected/9a767b5d-c7e1-4178-8793-4bd9f4352745-kube-api-access-x2vqp\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-systemd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238883    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-lib-modules\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238899    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-kubelet\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238912    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-netd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238943    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.239561 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238960    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-multus\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238965    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-cnibin\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238963    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de66365-30cc-4adc-ad1b-4a6778246af8-tmp-dir\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.238995    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-system-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.239003    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239027    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-cnibin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239069    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-host\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.239078    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:13.739057057 +0000 UTC m=+3.122900527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239119    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-systemd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjfw\" (UniqueName: \"kubernetes.io/projected/66f7168c-a3bc-43e5-a198-b3551b33b770-kube-api-access-fjjfw\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239209    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-run\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239221    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-env-overrides\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-tuned\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239259    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-netns\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239287    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-multus-certs\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239332    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.240423 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239343    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9de66365-30cc-4adc-ad1b-4a6778246af8-hosts-file\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239332    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-netd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239288    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9de66365-30cc-4adc-ad1b-4a6778246af8-hosts-file\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239382    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-hostroot\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239404    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-kubelet\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysconfig\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239432    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239456    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a767b5d-c7e1-4178-8793-4bd9f4352745-iptables-alerter-script\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239483    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239513    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-etc-kubernetes\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239527    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-slash\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.239607    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.239654    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret podName:bce76cad-b673-431c-b6a4-40298db25172 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:13.739636813 +0000 UTC m=+3.123480267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret") pod "global-pull-secret-syncer-rtd54" (UID: "bce76cad-b673-431c-b6a4-40298db25172") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239742    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-run\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239744    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-multus-daemon-config\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239762    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-script-lib\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239801    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-cni-binary-copy\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.241251 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239808    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-k8s-cni-cncf-io\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239819    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-netns\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-bin\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239879    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysconfig\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239884    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-device-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239920    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-tmp\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239927    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-device-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239947    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4bs\" (UniqueName: \"kubernetes.io/projected/8046eaad-8e51-49d2-beac-8f6be5170f4d-kube-api-access-sh4bs\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239963    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-bin\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.239974    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a767b5d-c7e1-4178-8793-4bd9f4352745-host-slash\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-dbus\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240029    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240056    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4lv\" (UniqueName: \"kubernetes.io/projected/63498c86-702e-47d4-8002-0693eb34b05e-kube-api-access-rv4lv\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-sys\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240098    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a767b5d-c7e1-4178-8793-4bd9f4352745-iptables-alerter-script\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-var-lib-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-node-log\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.242476 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240175    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-dbus\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-config\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240204    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240380    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-socket-dir-parent\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-bin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240453    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-slash\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240494    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-sys-fs\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240552    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240600    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-sys\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240620    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-var-lib-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240644    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-config\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240662    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240647    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-sys-fs\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240714    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a767b5d-c7e1-4178-8793-4bd9f4352745-host-slash\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240719    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-socket-dir-parent\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.243406 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-systemd-units\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240752    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-bin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240763    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-etc-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240764    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-kubelet\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240903    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-node-log\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240923    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-systemd-units\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240923    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-ovn\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240941    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-etc-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240980    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-ovn\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.240985    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-socket-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241029    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-registration-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241057    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-system-cni-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241122    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-system-cni-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rvh\" (UniqueName: \"kubernetes.io/projected/060b1735-396e-43d2-940d-e5b9bd3ec06b-kube-api-access-49rvh\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241150    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-socket-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241183    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-log-socket\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.244270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241205    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-registration-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241279    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63498c86-702e-47d4-8002-0693eb34b05e-ovn-node-metrics-cert\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241318    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241347    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/940bd241-206c-44c1-9c91-b6eb875bce7c-konnectivity-ca\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-netns\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241497    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-netns\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241534    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241691    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.241764    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.242033    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-log-socket\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.242284    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/940bd241-206c-44c1-9c91-b6eb875bce7c-konnectivity-ca\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.242428    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/940bd241-206c-44c1-9c91-b6eb875bce7c-agent-certs\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.242446    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-tmp\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.242887    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-tuned\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.245234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.245019    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63498c86-702e-47d4-8002-0693eb34b05e-ovn-node-metrics-cert\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.246343 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.246253    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" event={"ID":"2c7f2093544cf1327cc6ce2b1dd9d362","Type":"ContainerStarted","Data":"65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42"}
Oct 30 03:02:13.247659 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.247628    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"4be6db401bef7646097e7da845f451bd","Type":"ContainerStarted","Data":"b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b"}
Oct 30 03:02:13.262118 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.262082    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vqp\" (UniqueName: \"kubernetes.io/projected/9a767b5d-c7e1-4178-8793-4bd9f4352745-kube-api-access-x2vqp\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.262221 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.262210    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cj5d\" (UniqueName: \"kubernetes.io/projected/9de66365-30cc-4adc-ad1b-4a6778246af8-kube-api-access-5cj5d\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.265397 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.264946    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjfw\" (UniqueName: \"kubernetes.io/projected/66f7168c-a3bc-43e5-a198-b3551b33b770-kube-api-access-fjjfw\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.265397 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.264946    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkx9\" (UniqueName: \"kubernetes.io/projected/fdce563b-9436-449d-8cd6-159e7da40143-kube-api-access-zjkx9\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.265397 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.265149    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:02:13.265397 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.265179    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:02:13.265397 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.265194    2586 projected.go:196] Error preparing data for projected volume kube-api-access-xssbw for pod openshift-network-diagnostics/network-check-target-6ssh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:13.265397 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.265272    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw podName:3790bf39-be15-4435-8e27-f59fd78ce1af nodeName:}" failed. No retries permitted until 2026-10-30 03:02:13.76525266 +0000 UTC m=+3.149096135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xssbw" (UniqueName: "kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw") pod "network-check-target-6ssh4" (UID: "3790bf39-be15-4435-8e27-f59fd78ce1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:13.266118 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.266092    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4lv\" (UniqueName: \"kubernetes.io/projected/63498c86-702e-47d4-8002-0693eb34b05e-kube-api-access-rv4lv\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.268311 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.268281    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rvh\" (UniqueName: \"kubernetes.io/projected/060b1735-396e-43d2-940d-e5b9bd3ec06b-kube-api-access-49rvh\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.270720 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.270697    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4bs\" (UniqueName: \"kubernetes.io/projected/8046eaad-8e51-49d2-beac-8f6be5170f4d-kube-api-access-sh4bs\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.273634 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.273614    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8sr\" (UniqueName: \"kubernetes.io/projected/e00a6bb6-263b-40c7-977d-11511832d40a-kube-api-access-jd8sr\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.341602 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.341557    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:13.351094 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.351069    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:13.352828 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.352799    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da977d0_2885_4889_a18f_9038e7e11272.slice/crio-9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1 WatchSource:0}: Error finding container 9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1: Status 404 returned error can't find the container with id 9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1
Oct 30 03:02:13.363219 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.363185    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940bd241_206c_44c1_9c91_b6eb875bce7c.slice/crio-8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a WatchSource:0}: Error finding container 8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a: Status 404 returned error can't find the container with id 8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a
Oct 30 03:02:13.367369 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.367330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:13.374806 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.374635    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:13.383719 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.383696    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:13.384559 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.384530    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de66365_30cc_4adc_ad1b_4a6778246af8.slice/crio-9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18 WatchSource:0}: Error finding container 9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18: Status 404 returned error can't find the container with id 9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18
Oct 30 03:02:13.393233 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.392870    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-czcg4"
Oct 30 03:02:13.397125 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.397103    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a767b5d_c7e1_4178_8793_4bd9f4352745.slice/crio-41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f WatchSource:0}: Error finding container 41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f: Status 404 returned error can't find the container with id 41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f
Oct 30 03:02:13.404821 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.404798    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:13.415503 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.415473    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdce563b_9436_449d_8cd6_159e7da40143.slice/crio-6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e WatchSource:0}: Error finding container 6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e: Status 404 returned error can't find the container with id 6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e
Oct 30 03:02:13.429086 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.428917    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:13.440482 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.440460    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f7168c_a3bc_43e5_a198_b3551b33b770.slice/crio-bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b WatchSource:0}: Error finding container bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b: Status 404 returned error can't find the container with id bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b
Oct 30 03:02:13.442538 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.442273    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:13.452920 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:13.452894    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63498c86_702e_47d4_8002_0693eb34b05e.slice/crio-eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5 WatchSource:0}: Error finding container eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5: Status 404 returned error can't find the container with id eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5
Oct 30 03:02:13.744004 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.743919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:13.744004 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.743997    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:13.744209 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.744084    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:13.744209 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.744117    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:13.744209 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.744159    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret podName:bce76cad-b673-431c-b6a4-40298db25172 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:14.74413825 +0000 UTC m=+4.127981726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret") pod "global-pull-secret-syncer-rtd54" (UID: "bce76cad-b673-431c-b6a4-40298db25172") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:13.744209 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.744175    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:14.744167303 +0000 UTC m=+4.128010756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:13.844625 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:13.844589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:13.844786 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.844769    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:02:13.844859 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.844793    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:02:13.844859 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.844808    2586 projected.go:196] Error preparing data for projected volume kube-api-access-xssbw for pod openshift-network-diagnostics/network-check-target-6ssh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:13.844964 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:13.844867    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw podName:3790bf39-be15-4435-8e27-f59fd78ce1af nodeName:}" failed. No retries permitted until 2026-10-30 03:02:14.844847389 +0000 UTC m=+4.228690858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xssbw" (UniqueName: "kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw") pod "network-check-target-6ssh4" (UID: "3790bf39-be15-4435-8e27-f59fd78ce1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:14.088097 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.087977    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:13 +0000 UTC" deadline="2028-03-28 01:21:41.491256975 +0000 UTC"
Oct 30 03:02:14.088097 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.088050    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12358h19m27.40321074s"
Oct 30 03:02:14.253611 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.253565    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljjrp" event={"ID":"8046eaad-8e51-49d2-beac-8f6be5170f4d","Type":"ContainerStarted","Data":"953e904785230de608f86351ca55847bbed66ac0fb7f3f20a32e86d10898e1f9"}
Oct 30 03:02:14.255301 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.255272    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5"}
Oct 30 03:02:14.258353 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.258328    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czcg4" event={"ID":"e00a6bb6-263b-40c7-977d-11511832d40a","Type":"ContainerStarted","Data":"41726340b5ec7f10590aab25a069173160ebecedd130e921538b865946ee2081"}
Oct 30 03:02:14.260138 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.260105    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qcvz" event={"ID":"9a767b5d-c7e1-4178-8793-4bd9f4352745","Type":"ContainerStarted","Data":"41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f"}
Oct 30 03:02:14.263877 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.263851    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2hl5n" event={"ID":"9de66365-30cc-4adc-ad1b-4a6778246af8","Type":"ContainerStarted","Data":"9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18"}
Oct 30 03:02:14.269991 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.269972    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sfv5t" event={"ID":"940bd241-206c-44c1-9c91-b6eb875bce7c","Type":"ContainerStarted","Data":"8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a"}
Oct 30 03:02:14.272338 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.271917    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5sbs" event={"ID":"9da977d0-2885-4889-a18f-9038e7e11272","Type":"ContainerStarted","Data":"9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1"}
Oct 30 03:02:14.278611 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.278206    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerStarted","Data":"bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b"}
Oct 30 03:02:14.299508 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.299473    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw" event={"ID":"fdce563b-9436-449d-8cd6-159e7da40143","Type":"ContainerStarted","Data":"6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e"}
Oct 30 03:02:14.752322 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.751584    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:14.752322 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.751665    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:14.752322 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.751789    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:14.752322 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.751854    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:16.751835576 +0000 UTC m=+6.135679037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:14.752322 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.752240    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:14.752322 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.752287    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret podName:bce76cad-b673-431c-b6a4-40298db25172 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:16.752272682 +0000 UTC m=+6.136116142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret") pod "global-pull-secret-syncer-rtd54" (UID: "bce76cad-b673-431c-b6a4-40298db25172") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:14.853220 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:14.852878    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:14.853220 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.853087    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:02:14.853220 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.853106    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:02:14.853220 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.853118    2586 projected.go:196] Error preparing data for projected volume kube-api-access-xssbw for pod openshift-network-diagnostics/network-check-target-6ssh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:14.853220 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:14.853179    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw podName:3790bf39-be15-4435-8e27-f59fd78ce1af nodeName:}" failed. No retries permitted until 2026-10-30 03:02:16.853159218 +0000 UTC m=+6.237002679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xssbw" (UniqueName: "kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw") pod "network-check-target-6ssh4" (UID: "3790bf39-be15-4435-8e27-f59fd78ce1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:15.088904 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.088786    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:13 +0000 UTC" deadline="2028-04-15 11:53:12.940428522 +0000 UTC"
Oct 30 03:02:15.088904 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.088857    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12800h50m57.851576412s"
Oct 30 03:02:15.241809 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.240862    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:15.241809 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:15.241007    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:15.241809 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.241429    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:15.241809 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:15.241549    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:15.241809 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.241696    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:15.241809 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:15.241774    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:15.314702 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.313691    2586 generic.go:356] "Generic (PLEG): container finished" podID="4be6db401bef7646097e7da845f451bd" containerID="bfe27021a8036afec71ad276215bfd1c60838152b1b61bcc07792c81ca049539" exitCode=0
Oct 30 03:02:15.314702 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.313836    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"4be6db401bef7646097e7da845f451bd","Type":"ContainerDied","Data":"bfe27021a8036afec71ad276215bfd1c60838152b1b61bcc07792c81ca049539"}
Oct 30 03:02:15.343697 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:15.343166    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" event={"ID":"2c7f2093544cf1327cc6ce2b1dd9d362","Type":"ContainerStarted","Data":"e4f20c083e03336efa24a8911bbf1bb98bd03cc7eb93c119f4ae9e50c1e62e52"}
Oct 30 03:02:16.393553 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:16.393413    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"4be6db401bef7646097e7da845f451bd","Type":"ContainerStarted","Data":"ce51db10fedc2c4d27ed20e681fb052a79d451e1dd7568b2735a7bc97d581ba0"}
Oct 30 03:02:16.431400 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:16.430619    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" podStartSLOduration=3.430594447 podStartE2EDuration="3.430594447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:16.429976023 +0000 UTC m=+5.813819501" watchObservedRunningTime="2026-10-30 03:02:16.430594447 +0000 UTC m=+5.814437924"
Oct 30 03:02:16.431400 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:16.430752    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" podStartSLOduration=3.430741465 podStartE2EDuration="3.430741465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:15.385759781 +0000 UTC m=+4.769603250" watchObservedRunningTime="2026-10-30 03:02:16.430741465 +0000 UTC m=+5.814584944"
Oct 30 03:02:16.770945 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:16.770848    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:16.770945 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:16.770929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:16.771190 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.771078    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:16.771190 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.771149    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret podName:bce76cad-b673-431c-b6a4-40298db25172 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:20.771127838 +0000 UTC m=+10.154971313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret") pod "global-pull-secret-syncer-rtd54" (UID: "bce76cad-b673-431c-b6a4-40298db25172") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:16.771617 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.771597    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:16.771735 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.771656    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:20.771639618 +0000 UTC m=+10.155483087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:16.872455 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:16.871852    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:16.872455 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.872041    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:02:16.872455 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.872061    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:02:16.872455 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.872071    2586 projected.go:196] Error preparing data for projected volume kube-api-access-xssbw for pod openshift-network-diagnostics/network-check-target-6ssh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:16.872455 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:16.872118    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw podName:3790bf39-be15-4435-8e27-f59fd78ce1af nodeName:}" failed. No retries permitted until 2026-10-30 03:02:20.872102819 +0000 UTC m=+10.255946287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xssbw" (UniqueName: "kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw") pod "network-check-target-6ssh4" (UID: "3790bf39-be15-4435-8e27-f59fd78ce1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:17.243135 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:17.242199    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:17.243135 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:17.242376    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:17.243135 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:17.242441    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:17.243135 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:17.242539    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:17.243135 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:17.242596    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:17.243135 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:17.242679    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:17.401443 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:17.401385    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="f5a62a8f17b141add81f3cec4d6e990bcebd217c7c7309468508144b6d70a1a1" exitCode=0
Oct 30 03:02:17.403228 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:17.403193    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerDied","Data":"f5a62a8f17b141add81f3cec4d6e990bcebd217c7c7309468508144b6d70a1a1"}
Oct 30 03:02:18.406574 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:18.406519    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sfv5t" event={"ID":"940bd241-206c-44c1-9c91-b6eb875bce7c","Type":"ContainerStarted","Data":"a8f77c97f6243dec3184b3b0070627cf599e40946d5145ee888b8b3e7bc47a97"}
Oct 30 03:02:19.241702 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.241664    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:19.241868 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:19.241826    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:19.242265 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.242240    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:19.242373 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:19.242349    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:19.242456 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.242440    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:19.242552 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:19.242531    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:19.415528 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.413712    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5sbs" event={"ID":"9da977d0-2885-4889-a18f-9038e7e11272","Type":"ContainerStarted","Data":"728725f42148f22814557f7d9ae1e127abf2dfcca036a1366fb9eae13efd7bf2"}
Oct 30 03:02:19.418594 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.418543    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw" event={"ID":"fdce563b-9436-449d-8cd6-159e7da40143","Type":"ContainerStarted","Data":"a7ded3e27caa8ace01375b72d2b9f466288d890b5833275873a436bc85551a03"}
Oct 30 03:02:19.450577 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.450523    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z5sbs" podStartSLOduration=2.374689323 podStartE2EDuration="7.450504379s" totalImagesPullingTime="5.075815056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:19.450053288 +0000 UTC m=+8.833896768" watchObservedRunningTime="2026-10-30 03:02:19.450504379 +0000 UTC m=+8.834347862"
Oct 30 03:02:19.451149 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:19.451111    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sfv5t" podStartSLOduration=3.363372887 podStartE2EDuration="7.451099863s" totalImagesPullingTime="4.087726976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:18.445493117 +0000 UTC m=+7.829336597" watchObservedRunningTime="2026-10-30 03:02:19.451099863 +0000 UTC m=+8.834943342"
Oct 30 03:02:20.805899 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:20.805863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:20.806302 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:20.805933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:20.806302 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.806039    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:20.806302 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.806115    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:28.8060833 +0000 UTC m=+18.189926753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:20.806302 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.806150    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:20.806302 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.806218    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret podName:bce76cad-b673-431c-b6a4-40298db25172 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:28.806191263 +0000 UTC m=+18.190034719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret") pod "global-pull-secret-syncer-rtd54" (UID: "bce76cad-b673-431c-b6a4-40298db25172") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:20.907917 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:20.907201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:20.907917 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.907421    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:02:20.907917 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.907465    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:02:20.907917 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.907482    2586 projected.go:196] Error preparing data for projected volume kube-api-access-xssbw for pod openshift-network-diagnostics/network-check-target-6ssh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:20.907917 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:20.907547    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw podName:3790bf39-be15-4435-8e27-f59fd78ce1af nodeName:}" failed. No retries permitted until 2026-10-30 03:02:28.907525938 +0000 UTC m=+18.291369409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xssbw" (UniqueName: "kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw") pod "network-check-target-6ssh4" (UID: "3790bf39-be15-4435-8e27-f59fd78ce1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:20.951142 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:20.950979    2586 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 03:02:21.224897 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.224742    2586 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:02:20.950998926Z","UUID":"a80ba5b7-0cff-42f2-ac77-9042f00646f9","Handler":null,"Name":"","Endpoint":""}
Oct 30 03:02:21.232503 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.227691    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 03:02:21.232503 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.232082    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 03:02:21.245242 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.245218    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:21.247345 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.245597    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:21.247345 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.245650    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:21.248095 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:21.248054    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:21.248826 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:21.248779    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:21.249043 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:21.248501    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:21.351171 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.350491    2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:21.351796 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.351604    2586 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:21.429172 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:21.429131    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw" event={"ID":"fdce563b-9436-449d-8cd6-159e7da40143","Type":"ContainerStarted","Data":"fab22846017c694a2c0c84e6317efd1fbc6bdce235dfb1c669e5d7ccffbd8017"}
Oct 30 03:02:22.434565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.434299    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2hl5n" event={"ID":"9de66365-30cc-4adc-ad1b-4a6778246af8","Type":"ContainerStarted","Data":"53c196bdea6c49ee83a7945cdd351a6c54bc3dddf13fc7221dcd445c1731fb52"}
Oct 30 03:02:22.436977 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.436946    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="1df978a25c2fc1c48b887497ffae6f3fa9136f00a8c931a690d4b83828a7c86f" exitCode=0
Oct 30 03:02:22.437097 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.437034    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerDied","Data":"1df978a25c2fc1c48b887497ffae6f3fa9136f00a8c931a690d4b83828a7c86f"}
Oct 30 03:02:22.441334 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.441306    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw" event={"ID":"fdce563b-9436-449d-8cd6-159e7da40143","Type":"ContainerStarted","Data":"22b22a70b5ea82450ea54025a23fb5f2eb61444645891d5bb7dd471a012702aa"}
Oct 30 03:02:22.443260 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.443234    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-czcg4" event={"ID":"e00a6bb6-263b-40c7-977d-11511832d40a","Type":"ContainerStarted","Data":"e1a4e92e648cf0fbd0390bc6c5b3491f1eefbc3f69ab5be09af85110b46bdd77"}
Oct 30 03:02:22.537397 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.537326    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw" podStartSLOduration=2.149113668 podStartE2EDuration="10.537306789s" totalImagesPullingTime="8.388193121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:22.53683041 +0000 UTC m=+11.920673888" watchObservedRunningTime="2026-10-30 03:02:22.537306789 +0000 UTC m=+11.921150269"
Oct 30 03:02:22.537628 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.537601    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2hl5n" podStartSLOduration=2.355142857 podStartE2EDuration="10.537592874s" totalImagesPullingTime="8.182450017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:22.495808028 +0000 UTC m=+11.879651508" watchObservedRunningTime="2026-10-30 03:02:22.537592874 +0000 UTC m=+11.921436352"
Oct 30 03:02:22.578113 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:22.578043    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-czcg4" podStartSLOduration=2.543694438 podStartE2EDuration="10.578023351s" totalImagesPullingTime="8.034328913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:22.577436204 +0000 UTC m=+11.961279703" watchObservedRunningTime="2026-10-30 03:02:22.578023351 +0000 UTC m=+11.961866830"
Oct 30 03:02:23.239903 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.239876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:23.240051 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:23.240001    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:23.240051 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.239876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:23.240128 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:23.240107    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:23.240170 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.240134    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:23.240215 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:23.240193    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:23.448241 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.448206    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="2b850a08644bd591473705115b3391950acc840759296c0c8b26aa5761d1c69c" exitCode=0
Oct 30 03:02:23.448613 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.448299    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerDied","Data":"2b850a08644bd591473705115b3391950acc840759296c0c8b26aa5761d1c69c"}
Oct 30 03:02:23.451814 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.451771    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qcvz" event={"ID":"9a767b5d-c7e1-4178-8793-4bd9f4352745","Type":"ContainerStarted","Data":"88d6702fcc155974a2bc716bd736aad859c83f9c0e03b7a2c104a72e7a57330a"}
Oct 30 03:02:23.520231 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:23.520171    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2qcvz" podStartSLOduration=3.012076485 podStartE2EDuration="11.520154955s" totalImagesPullingTime="8.50807847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:23.519850916 +0000 UTC m=+12.903694396" watchObservedRunningTime="2026-10-30 03:02:23.520154955 +0000 UTC m=+12.903998430"
Oct 30 03:02:24.454762 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:24.454729    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljjrp" event={"ID":"8046eaad-8e51-49d2-beac-8f6be5170f4d","Type":"ContainerStarted","Data":"5825a6955385364f14a91015e4aa294f20ecf8e9a1d03d438e679eb16b9b3da7"}
Oct 30 03:02:25.240732 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:25.240693    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:25.240732 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:25.240720    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:25.240964 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:25.240814    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:25.240964 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:25.240829    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:25.240964 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:25.240938    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:25.241114 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:25.241033    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:25.459996 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:25.459958    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="17f725d0431edd4fa903f2d99568c91b0e8315dbf937a08cf90591115c189166" exitCode=0
Oct 30 03:02:25.460573 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:25.460051    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerDied","Data":"17f725d0431edd4fa903f2d99568c91b0e8315dbf937a08cf90591115c189166"}
Oct 30 03:02:25.519412 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:25.519321    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ljjrp" podStartSLOduration=2.797571247 podStartE2EDuration="13.519307904s" totalImagesPullingTime="10.721736657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:24.489839932 +0000 UTC m=+13.873683407" watchObservedRunningTime="2026-10-30 03:02:25.519307904 +0000 UTC m=+14.903151465"
Oct 30 03:02:26.210506 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:26.210473    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:26.210695 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:26.210618    2586 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:02:26.211668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:26.211628    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:27.240557 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:27.240523    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:27.240557 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:27.240538    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:27.241140 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:27.240645    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:27.241140 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:27.240681    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:27.241140 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:27.240717    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:27.241140 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:27.240806    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:27.467961 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:27.467753    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerStarted","Data":"45b24d2c8a442535cc05b1c80bcf6403ee635040ae00c7b81a692243f94a0a48"}
Oct 30 03:02:28.471844 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:28.471805    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="45b24d2c8a442535cc05b1c80bcf6403ee635040ae00c7b81a692243f94a0a48" exitCode=0
Oct 30 03:02:28.472238 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:28.471848    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerDied","Data":"45b24d2c8a442535cc05b1c80bcf6403ee635040ae00c7b81a692243f94a0a48"}
Oct 30 03:02:28.871012 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:28.870844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:28.871109 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:28.871069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:28.871109 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.870999    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:28.871201 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.871148    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret podName:bce76cad-b673-431c-b6a4-40298db25172 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.871124656 +0000 UTC m=+34.254968120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret") pod "global-pull-secret-syncer-rtd54" (UID: "bce76cad-b673-431c-b6a4-40298db25172") : object "kube-system"/"original-pull-secret" not registered
Oct 30 03:02:28.871201 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.871194    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:28.871288 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.871250    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.871235225 +0000 UTC m=+34.255078692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 03:02:28.972299 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:28.972242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:28.972414 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.972361    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 03:02:28.972414 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.972373    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 03:02:28.972414 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.972381    2586 projected.go:196] Error preparing data for projected volume kube-api-access-xssbw for pod openshift-network-diagnostics/network-check-target-6ssh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:28.972557 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:28.972418    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw podName:3790bf39-be15-4435-8e27-f59fd78ce1af nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.972407469 +0000 UTC m=+34.356250935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xssbw" (UniqueName: "kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw") pod "network-check-target-6ssh4" (UID: "3790bf39-be15-4435-8e27-f59fd78ce1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 03:02:29.239993 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.239902    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:29.240167 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.239916    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:29.240167 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.240044    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:29.240167 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:29.240020    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:29.240167 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:29.240109    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:29.240391 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:29.240177    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:29.477037 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.476999    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"34cf7f07c70bcf8e4cbe81b14528198380502f0522000b4aa83a862a12f18457"}
Oct 30 03:02:29.477492 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.477067    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"4a0dcbc55c6f4027e381d00627fe272d155f094496600c32cddb3771a7092f88"}
Oct 30 03:02:29.477492 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.477101    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"9a8613dea0a860b7fc0be26f1e2de61216cccd4da78de56532a8f8943235afae"}
Oct 30 03:02:29.477492 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.477134    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"6b4ace9eb977d18f21bfefa82b4765101c81435f99d91bc7d9ef36f870d36d4c"}
Oct 30 03:02:29.477492 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.477159    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"729c31e00e50baba5024c7f8db3236013e5d76bb435dafe31d040b12918ee4b6"}
Oct 30 03:02:29.477492 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.477193    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"c153be4b9cf5aedb5ed9d4d529d2e9485b3229e220fb8e6d4974249caa026031"}
Oct 30 03:02:29.479908 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.479881    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="0848a71801dc5f5e6fcb668ce792c38b97670cd3bc947f02f7eb3671770e6720" exitCode=0
Oct 30 03:02:29.480014 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:29.479920    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerDied","Data":"0848a71801dc5f5e6fcb668ce792c38b97670cd3bc947f02f7eb3671770e6720"}
Oct 30 03:02:30.484813 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:30.484775    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" event={"ID":"66f7168c-a3bc-43e5-a198-b3551b33b770","Type":"ContainerStarted","Data":"6a15cab604cfbad17938233fdf0f32b214692c34b5103fc063e79c6ff86f15ea"}
Oct 30 03:02:30.542938 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:30.542895    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8wqsd" podStartSLOduration=7.745391657 podStartE2EDuration="18.542882009s" totalImagesPullingTime="10.797490352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:30.53258363 +0000 UTC m=+19.916427103" watchObservedRunningTime="2026-10-30 03:02:30.542882009 +0000 UTC m=+19.926725515"
Oct 30 03:02:31.240950 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:31.240863    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:31.241088 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:31.240961    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:31.241088 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:31.241065    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:31.241227 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:31.241200    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:31.241276 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:31.241214    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:31.241321 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:31.241304    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:31.490048 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:31.490011    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"fde80d743363835194d458629110bf2f74f717aef2aebdf8ecddced00b4a8045"}
Oct 30 03:02:33.240607 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.240574    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:33.241164 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.240579    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:33.241164 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:33.240712    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:33.241164 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.240728    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:33.241164 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:33.240812    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:33.241164 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:33.240873    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:33.497551 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.497240    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" event={"ID":"63498c86-702e-47d4-8002-0693eb34b05e","Type":"ContainerStarted","Data":"817d972c6e8890cfe38734a35723888b58dbd15a27f4cf6b46e7db96e1f5e74e"}
Oct 30 03:02:33.497712 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.497560    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:33.497712 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.497607    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:33.512700 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.512678    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:33.512808 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.512798    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:33.576619 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:33.576576    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn" podStartSLOduration=6.414976361 podStartE2EDuration="21.576564604s" totalImagesPullingTime="15.161588243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:33.566028733 +0000 UTC m=+22.949872206" watchObservedRunningTime="2026-10-30 03:02:33.576564604 +0000 UTC m=+22.960408079"
Oct 30 03:02:34.083960 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:34.083931    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:35.152466 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:35.152376    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rtd54"]
Oct 30 03:02:35.152817 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:35.152515    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:35.152817 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:35.152596    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:35.155239 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:35.155216    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ssh4"]
Oct 30 03:02:35.155340 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:35.155299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:35.155383 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:35.155363    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:35.162357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:35.162334    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z74kk"]
Oct 30 03:02:35.162458 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:35.162441    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:35.162534 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:35.162516    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:36.240343 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:36.240312    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:36.240716 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:36.240447    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:37.240339 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:37.240311    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:37.240494 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:37.240311    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:37.240494 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:37.240419    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:37.240820 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:37.240540    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:38.240478 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:38.240450    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:38.240703 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:38.240548    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:39.239949 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:39.239921    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:39.240092 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:39.240016    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rtd54" podUID="bce76cad-b673-431c-b6a4-40298db25172"
Oct 30 03:02:39.240152 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:39.240088    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:39.240206 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:39.240167    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ssh4" podUID="3790bf39-be15-4435-8e27-f59fd78ce1af"
Oct 30 03:02:40.240356 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.240328    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:40.240720 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:40.240439    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z74kk" podUID="060b1735-396e-43d2-940d-e5b9bd3ec06b"
Oct 30 03:02:40.920725 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.920703    2586 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeReady"
Oct 30 03:02:40.920836 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.920824    2586 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 03:02:40.965456 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.965411    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cd96bbd68-84shk"]
Oct 30 03:02:40.967881 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.967863    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:40.968724 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.968704    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zwfb4"]
Oct 30 03:02:40.971936 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.971920    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gt4dd"]
Oct 30 03:02:40.972124 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.972107    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:40.974992 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.974973    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ptzxj"]
Oct 30 03:02:40.975361 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.975076    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:40.978038 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.978021    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"]
Oct 30 03:02:40.978183 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.978169    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:40.981558 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.981543    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"]
Oct 30 03:02:40.981657 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.981643    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:40.984313 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.984296    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"]
Oct 30 03:02:40.984405 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.984391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:40.988001 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.987967    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"]
Oct 30 03:02:40.988090 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.988069    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:40.989055 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.988951    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 03:02:40.989556 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.989539    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 03:02:40.990876 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.990855    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"]
Oct 30 03:02:40.990996 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.990952    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:40.993656 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.993640    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-klw22"]
Oct 30 03:02:40.993781 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.993767    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:40.996552 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.996537    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59ddf86b9f-2wfp9"]
Oct 30 03:02:40.996720 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.996706    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:40.999556 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:40.999534    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"]
Oct 30 03:02:41.000797 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.000777    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.003188 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.003169    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lrntf"]
Oct 30 03:02:41.003292 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.003278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.005990 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.005970    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"]
Oct 30 03:02:41.006106 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.006080    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.008701 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.008686    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"]
Oct 30 03:02:41.008802 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.008788    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.011501 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.011483    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:41.011859 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.011840    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wcx8m\""
Oct 30 03:02:41.011965 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.011843    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.011965 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.011885    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.012863 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.012845    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 03:02:41.012987 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.012973    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zwfb4"]
Oct 30 03:02:41.014627 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.014607    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ptzxj"]
Oct 30 03:02:41.020498 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.020480    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gt4dd"]
Oct 30 03:02:41.027603 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.027564    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd96bbd68-84shk"]
Oct 30 03:02:41.032499 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.029636    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ptqz4"]
Oct 30 03:02:41.036446 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.035321    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cfphv"]
Oct 30 03:02:41.036446 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.035862    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.039693 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.039677    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"]
Oct 30 03:02:41.039845 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.039832    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-klw22"]
Oct 30 03:02:41.039951 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.039942    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"]
Oct 30 03:02:41.040044 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.040034    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"]
Oct 30 03:02:41.040212 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.040191    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.040381 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.040359    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"]
Oct 30 03:02:41.047005 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.046977    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"]
Oct 30 03:02:41.047996 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.047978    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59ddf86b9f-2wfp9"]
Oct 30 03:02:41.049795 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.049772    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"]
Oct 30 03:02:41.063103 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.062602    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"]
Oct 30 03:02:41.063619 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.063590    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ptqz4"]
Oct 30 03:02:41.064543 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.064525    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lrntf"]
Oct 30 03:02:41.065572 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065483    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"]
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065755    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplzq\" (UniqueName: \"kubernetes.io/projected/74e50c74-23f8-46b7-98e6-6ac103daba15-kube-api-access-nplzq\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065784    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-config\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065819    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqq7j\" (UniqueName: \"kubernetes.io/projected/160c1f29-a378-43d0-97d8-be1cc643e2c2-kube-api-access-bqq7j\") pod \"kas-connection-checker-685895c7c9-zwfb4\" (UID: \"160c1f29-a378-43d0-97d8-be1cc643e2c2\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065845    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e29e28e-149b-4e87-af1c-a1ab78aa1969-config\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065894    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hps\" (UniqueName: \"kubernetes.io/projected/1e29e28e-149b-4e87-af1c-a1ab78aa1969-kube-api-access-p2hps\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065935    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065958    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e29e28e-149b-4e87-af1c-a1ab78aa1969-serving-cert\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.065985    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066006    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066029    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74e50c74-23f8-46b7-98e6-6ac103daba15-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/af46b57a-5155-4ddd-90d8-bf2eb36f8319-kube-api-access-8z5dm\") pod \"kas-connection-checker-685895c7c9-gt4dd\" (UID: \"af46b57a-5155-4ddd-90d8-bf2eb36f8319\") " pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066086    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-trusted-ca\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.070096 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066117    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066142    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5pzb\" (UniqueName: \"kubernetes.io/projected/b7114a74-db33-4c40-a401-030ffa957f21-kube-api-access-k5pzb\") pod \"volume-data-source-validator-58c9f5f46d-2mdnr\" (UID: \"b7114a74-db33-4c40-a401-030ffa957f21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066238    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-serving-cert\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066399    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066476    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066527    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fcp\" (UniqueName: \"kubernetes.io/projected/d1d1d308-d556-4dbc-bd3a-2ec2c838413a-kube-api-access-22fcp\") pod \"kas-connection-checker-685895c7c9-ptzxj\" (UID: \"d1d1d308-d556-4dbc-bd3a-2ec2c838413a\") " pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066698    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066790    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066831    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrj2\" (UniqueName: \"kubernetes.io/projected/bec84fda-07e9-469d-bec4-16e9586fed14-kube-api-access-kjrj2\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.066860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8549h\" (UniqueName: \"kubernetes.io/projected/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-kube-api-access-8549h\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.070882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.068002    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cfphv"]
Oct 30 03:02:41.087611 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.087591    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 03:02:41.088179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.088153    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 03:02:41.088780 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.088735    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.089214 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089196    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.089297 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089276    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tpt7c\""
Oct 30 03:02:41.089514 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089498    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 03:02:41.089682 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089667    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dt8fk\""
Oct 30 03:02:41.089768 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089748    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 03:02:41.089814 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089775    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 03:02:41.089922 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.089907    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 03:02:41.090127 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090110    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.090127 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090121    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 03:02:41.090315 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090301    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whjh9\""
Oct 30 03:02:41.090377 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090359    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.090462 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090400    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8g9pb\""
Oct 30 03:02:41.090522 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090481    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.090675 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090657    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 03:02:41.090785 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090766    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 03:02:41.090834 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090784    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.090882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090868    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.090929 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.090892    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 03:02:41.091192 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091175    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 03:02:41.091329 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091313    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 03:02:41.091565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091549    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r5bpm\""
Oct 30 03:02:41.091621 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091570    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cq8qc\""
Oct 30 03:02:41.091665 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091620    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.091717 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091685    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t77t7\""
Oct 30 03:02:41.091768 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091720    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.091817 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091786    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-njwkw\""
Oct 30 03:02:41.091905 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091890    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 03:02:41.091954 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091905    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.092054 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092037    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.092098 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092072    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.092143 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092099    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 03:02:41.092190 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092161    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.092234 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092204    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.092295 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092280    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 03:02:41.092339 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092309    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.092339 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092318    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 03:02:41.092438 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092390    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.092493 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.091553    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.092536 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092511    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 03:02:41.092635 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092617    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 03:02:41.092680 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092045    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.092732 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092722    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 03:02:41.092821 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092796    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 03:02:41.092878 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092823    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r8pw4\""
Oct 30 03:02:41.092950 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.092811    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 03:02:41.093789 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.093764    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-scdl7\""
Oct 30 03:02:41.093789 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.093781    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 03:02:41.093789 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.093782    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 03:02:41.093789 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.093812    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgj97\""
Oct 30 03:02:41.094027 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.093845    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vnb2g\""
Oct 30 03:02:41.098242 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.098226    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 03:02:41.100124 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.100106    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 03:02:41.100757 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.100733    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 03:02:41.167384 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167358    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e29e28e-149b-4e87-af1c-a1ab78aa1969-config\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167411    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-config\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167444    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e29e28e-149b-4e87-af1c-a1ab78aa1969-serving-cert\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167463    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqq7j\" (UniqueName: \"kubernetes.io/projected/160c1f29-a378-43d0-97d8-be1cc643e2c2-kube-api-access-bqq7j\") pod \"kas-connection-checker-685895c7c9-zwfb4\" (UID: \"160c1f29-a378-43d0-97d8-be1cc643e2c2\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bbg\" (UniqueName: \"kubernetes.io/projected/3e49f31c-2187-47b3-89d9-1e76ec6f883a-kube-api-access-w4bbg\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167498    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hps\" (UniqueName: \"kubernetes.io/projected/1e29e28e-149b-4e87-af1c-a1ab78aa1969-kube-api-access-p2hps\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.167513 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167514    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167537    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167580    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-tmp\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krn5\" (UniqueName: \"kubernetes.io/projected/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-kube-api-access-8krn5\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.167964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168001    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-stats-auth\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168055    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168092    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-tmp-dir\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168155    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168196    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fcp\" (UniqueName: \"kubernetes.io/projected/d1d1d308-d556-4dbc-bd3a-2ec2c838413a-kube-api-access-22fcp\") pod \"kas-connection-checker-685895c7c9-ptzxj\" (UID: \"d1d1d308-d556-4dbc-bd3a-2ec2c838413a\") " pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168227    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e29e28e-149b-4e87-af1c-a1ab78aa1969-config\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.168290 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168280    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8549h\" (UniqueName: \"kubernetes.io/projected/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-kube-api-access-8549h\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168331    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-config\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168337    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168405    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168450    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmg2\" (UniqueName: \"kubernetes.io/projected/f6a644f9-7b06-4072-8052-704244dd5a8d-kube-api-access-qbmg2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168517    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168517    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168541    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168606    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6a644f9-7b06-4072-8052-704244dd5a8d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.168619    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168642    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2mp\" (UniqueName: \"kubernetes.io/projected/29f57388-aede-46d4-9af1-075947d71aa3-kube-api-access-9k2mp\") pod \"network-check-source-6f78d59f99-zn99v\" (UID: \"29f57388-aede-46d4-9af1-075947d71aa3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.168680    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.668664103 +0000 UTC m=+31.052507570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168700    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplzq\" (UniqueName: \"kubernetes.io/projected/74e50c74-23f8-46b7-98e6-6ac103daba15-kube-api-access-nplzq\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168728    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.169076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a644f9-7b06-4072-8052-704244dd5a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168789    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168815    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-snapshots\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168838    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-config-volume\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168862    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-default-certificate\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168897    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168926    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168952    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168954    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.168960    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.168977    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e49f31c-2187-47b3-89d9-1e76ec6f883a-serving-cert\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.169032    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.669014659 +0000 UTC m=+31.052858126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-trusted-ca\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169153    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169218    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba135d3a-8995-4181-a315-51ae998bba50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.171211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169264    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pzb\" (UniqueName: \"kubernetes.io/projected/b7114a74-db33-4c40-a401-030ffa957f21-kube-api-access-k5pzb\") pod \"volume-data-source-validator-58c9f5f46d-2mdnr\" (UID: \"b7114a74-db33-4c40-a401-030ffa957f21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169348    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-serving-cert\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169477    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169504    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35010af-87bc-4a97-81b6-6803ffd77e25-cert\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169566    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km82f\" (UniqueName: \"kubernetes.io/projected/1569b662-064a-4998-b332-03c6ed48e312-kube-api-access-km82f\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169636    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169666    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrj2\" (UniqueName: \"kubernetes.io/projected/bec84fda-07e9-469d-bec4-16e9586fed14-kube-api-access-kjrj2\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/af46b57a-5155-4ddd-90d8-bf2eb36f8319-kube-api-access-8z5dm\") pod \"kas-connection-checker-685895c7c9-gt4dd\" (UID: \"af46b57a-5155-4ddd-90d8-bf2eb36f8319\") " pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169756    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-service-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169792    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169822    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7ct\" (UniqueName: \"kubernetes.io/projected/a35010af-87bc-4a97-81b6-6803ffd77e25-kube-api-access-fz7ct\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.169854    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74e50c74-23f8-46b7-98e6-6ac103daba15-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.170321    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-trusted-ca\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.171933 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.170372    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.170374    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.170628    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.170646    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.170679    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74e50c74-23f8-46b7-98e6-6ac103daba15-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.170694    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.670684229 +0000 UTC m=+31.054527684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.172215    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.172233    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e29e28e-149b-4e87-af1c-a1ab78aa1969-serving-cert\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.172357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.172216    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-serving-cert\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.172611 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.172455    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.185070 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.185053    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fcp\" (UniqueName: \"kubernetes.io/projected/d1d1d308-d556-4dbc-bd3a-2ec2c838413a-kube-api-access-22fcp\") pod \"kas-connection-checker-685895c7c9-ptzxj\" (UID: \"d1d1d308-d556-4dbc-bd3a-2ec2c838413a\") " pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:41.188780 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.188754    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqq7j\" (UniqueName: \"kubernetes.io/projected/160c1f29-a378-43d0-97d8-be1cc643e2c2-kube-api-access-bqq7j\") pod \"kas-connection-checker-685895c7c9-zwfb4\" (UID: \"160c1f29-a378-43d0-97d8-be1cc643e2c2\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:41.188977 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.188952    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8549h\" (UniqueName: \"kubernetes.io/projected/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-kube-api-access-8549h\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.190223 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.189856    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.190223 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.190202    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.190831 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.190810    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hps\" (UniqueName: \"kubernetes.io/projected/1e29e28e-149b-4e87-af1c-a1ab78aa1969-kube-api-access-p2hps\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.190930 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.190888    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pzb\" (UniqueName: \"kubernetes.io/projected/b7114a74-db33-4c40-a401-030ffa957f21-kube-api-access-k5pzb\") pod \"volume-data-source-validator-58c9f5f46d-2mdnr\" (UID: \"b7114a74-db33-4c40-a401-030ffa957f21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:41.190930 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.190890    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrj2\" (UniqueName: \"kubernetes.io/projected/bec84fda-07e9-469d-bec4-16e9586fed14-kube-api-access-kjrj2\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.191235 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.191214    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplzq\" (UniqueName: \"kubernetes.io/projected/74e50c74-23f8-46b7-98e6-6ac103daba15-kube-api-access-nplzq\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.192106 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.192086    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/af46b57a-5155-4ddd-90d8-bf2eb36f8319-kube-api-access-8z5dm\") pod \"kas-connection-checker-685895c7c9-gt4dd\" (UID: \"af46b57a-5155-4ddd-90d8-bf2eb36f8319\") " pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:41.240080 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.240054    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:41.240080 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.240086    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:41.252806 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.252788    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-69wt9\""
Oct 30 03:02:41.255393 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.255376    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-shnp4\""
Oct 30 03:02:41.270216 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.270307 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270226    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmg2\" (UniqueName: \"kubernetes.io/projected/f6a644f9-7b06-4072-8052-704244dd5a8d-kube-api-access-qbmg2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.270307 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270266    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.270385 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270314    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6a644f9-7b06-4072-8052-704244dd5a8d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.270385 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2mp\" (UniqueName: \"kubernetes.io/projected/29f57388-aede-46d4-9af1-075947d71aa3-kube-api-access-9k2mp\") pod \"network-check-source-6f78d59f99-zn99v\" (UID: \"29f57388-aede-46d4-9af1-075947d71aa3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:41.270385 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270370    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.270524 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.270387    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:41.270524 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270417    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a644f9-7b06-4072-8052-704244dd5a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.270524 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.270514    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.770472504 +0000 UTC m=+31.154315967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:41.270668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-snapshots\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.270668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-config-volume\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.270668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270565    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-default-certificate\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.270668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270583    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.270668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270603    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.270668 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270619    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e49f31c-2187-47b3-89d9-1e76ec6f883a-serving-cert\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.270866 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.270866 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270725    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba135d3a-8995-4181-a315-51ae998bba50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.270866 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270787    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.270866 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270814    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35010af-87bc-4a97-81b6-6803ffd77e25-cert\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.270866 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270841    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km82f\" (UniqueName: \"kubernetes.io/projected/1569b662-064a-4998-b332-03c6ed48e312-kube-api-access-km82f\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.271076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.270978    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:41.271076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.271026    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.771012266 +0000 UTC m=+31.154855732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:41.271174 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271109    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6a644f9-7b06-4072-8052-704244dd5a8d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.271174 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.271270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271166    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-config-volume\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.271270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271195    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-service-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.271270 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271227    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7ct\" (UniqueName: \"kubernetes.io/projected/a35010af-87bc-4a97-81b6-6803ffd77e25-kube-api-access-fz7ct\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.271386 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271281    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bbg\" (UniqueName: \"kubernetes.io/projected/3e49f31c-2187-47b3-89d9-1e76ec6f883a-kube-api-access-w4bbg\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.271386 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271311    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.271386 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271335    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.271565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271377    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-tmp\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.271565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271448    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krn5\" (UniqueName: \"kubernetes.io/projected/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-kube-api-access-8krn5\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.271565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271486    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.271565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271512    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-stats-auth\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.271565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.271565 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271565    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.272188 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.272166    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.772154152 +0000 UTC m=+31.155997604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:41.272702 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.272651    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba135d3a-8995-4181-a315-51ae998bba50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.273183 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.273031    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.273183 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.273093    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-snapshots\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.273618 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.270615    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.273992 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.273949    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-service-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.274353 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.274322    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.274808 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.274787    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:41.274924 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.274810    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:41.274924 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.274887    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.774874002 +0000 UTC m=+31.158717455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:41.275214 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.275194    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-tmp\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.275677 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.275655    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.277820 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.277382    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:41.277820 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.277460    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:41.777443301 +0000 UTC m=+31.161286768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:41.277820 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.277786    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-tmp-dir\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.282361 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.277794    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e49f31c-2187-47b3-89d9-1e76ec6f883a-serving-cert\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.282535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.277812    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a644f9-7b06-4072-8052-704244dd5a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.282535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.271589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-tmp-dir\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.282535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.278569    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.282535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.279534    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.282535 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.279552    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-default-certificate\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.283405 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.283386    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35010af-87bc-4a97-81b6-6803ffd77e25-cert\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.284300 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.284278    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-stats-auth\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.287235 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.287208    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:41.289878 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.289856    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2mp\" (UniqueName: \"kubernetes.io/projected/29f57388-aede-46d4-9af1-075947d71aa3-kube-api-access-9k2mp\") pod \"network-check-source-6f78d59f99-zn99v\" (UID: \"29f57388-aede-46d4-9af1-075947d71aa3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:41.289975 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.289922    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmg2\" (UniqueName: \"kubernetes.io/projected/f6a644f9-7b06-4072-8052-704244dd5a8d-kube-api-access-qbmg2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.295041 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.294857    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:41.295525 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.295497    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.295683 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.295661    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km82f\" (UniqueName: \"kubernetes.io/projected/1569b662-064a-4998-b332-03c6ed48e312-kube-api-access-km82f\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.297065 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.297039    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krn5\" (UniqueName: \"kubernetes.io/projected/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-kube-api-access-8krn5\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.298902 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.298879    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.298973 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.298964    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7ct\" (UniqueName: \"kubernetes.io/projected/a35010af-87bc-4a97-81b6-6803ffd77e25-kube-api-access-fz7ct\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.299103 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.299076    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bbg\" (UniqueName: \"kubernetes.io/projected/3e49f31c-2187-47b3-89d9-1e76ec6f883a-kube-api-access-w4bbg\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.300737 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.300711    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:41.317211 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.317186    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:41.327003 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.324238    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:41.342219 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.342131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:41.353946 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.353922    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 03:02:41.357314 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.357283    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:41.381023 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.380716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:41.424945 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.423584    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:41.424945 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.423878    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:41.615054 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.614983    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zwfb4"]
Oct 30 03:02:41.616612 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.616345    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ptzxj"]
Oct 30 03:02:41.618243 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.618208    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gt4dd"]
Oct 30 03:02:41.619040 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.618986    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c1f29_a378_43d0_97d8_be1cc643e2c2.slice/crio-29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e WatchSource:0}: Error finding container 29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e: Status 404 returned error can't find the container with id 29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e
Oct 30 03:02:41.619692 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.619667    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d1d308_d556_4dbc_bd3a_2ec2c838413a.slice/crio-58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5 WatchSource:0}: Error finding container 58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5: Status 404 returned error can't find the container with id 58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5
Oct 30 03:02:41.622562 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.621483    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf46b57a_5155_4ddd_90d8_bf2eb36f8319.slice/crio-d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc WatchSource:0}: Error finding container d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc: Status 404 returned error can't find the container with id d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc
Oct 30 03:02:41.650842 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.650818    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"]
Oct 30 03:02:41.654556 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.654534    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7114a74_db33_4c40_a401_030ffa957f21.slice/crio-3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3 WatchSource:0}: Error finding container 3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3: Status 404 returned error can't find the container with id 3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3
Oct 30 03:02:41.664657 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.663930    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"]
Oct 30 03:02:41.672962 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.672934    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a65983f_48e3_4e8f_a576_2110dfbb2ec2.slice/crio-dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd WatchSource:0}: Error finding container dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd: Status 404 returned error can't find the container with id dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.687855    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.687931    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.687995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688147    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688160    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688206    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.688189601 +0000 UTC m=+32.072033070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688270    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688304    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.688293618 +0000 UTC m=+32.072137086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688364    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:41.688435 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.688394    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.68838478 +0000 UTC m=+32.072228247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:41.694378 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.694105    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-klw22"]
Oct 30 03:02:41.707966 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.707938    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e49f31c_2187_47b3_89d9_1e76ec6f883a.slice/crio-54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830 WatchSource:0}: Error finding container 54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830: Status 404 returned error can't find the container with id 54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830
Oct 30 03:02:41.717310 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.717286    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"]
Oct 30 03:02:41.747418 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.747393    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"]
Oct 30 03:02:41.755551 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.755523    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a644f9_7b06_4072_8052_704244dd5a8d.slice/crio-146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54 WatchSource:0}: Error finding container 146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54: Status 404 returned error can't find the container with id 146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54
Oct 30 03:02:41.781484 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.781453    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"]
Oct 30 03:02:41.784031 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.784010    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ptqz4"]
Oct 30 03:02:41.784552 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.784531    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f57388_aede_46d4_9af1_075947d71aa3.slice/crio-fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f WatchSource:0}: Error finding container fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f: Status 404 returned error can't find the container with id fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f
Oct 30 03:02:41.787274 ip-10-0-135-226 kubenswrapper[2586]: W1030 03:02:41.787254    2586 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35010af_87bc_4a97_81b6_6803ffd77e25.slice/crio-77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240 WatchSource:0}: Error finding container 77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240: Status 404 returned error can't find the container with id 77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240
Oct 30 03:02:41.788472 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.788448    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:41.788550 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.788534    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.788610 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.788595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:41.788656 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788601    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:41.788656 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.788642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:41.788749 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788656    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:41.788749 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788660    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:41.788749 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788696    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:41.788749 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788705    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.788685582 +0000 UTC m=+32.172529056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:41.789033 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788767    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.788751588 +0000 UTC m=+32.172595260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:41.789033 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:41.788765    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:41.789033 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788790    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.78878063 +0000 UTC m=+32.172624090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:41.789033 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788809    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.788800865 +0000 UTC m=+32.172644318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:41.789033 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788818    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:41.789033 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:41.788872    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:42.78885723 +0000 UTC m=+32.172700684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:42.240241 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.240209    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:42.253265 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.253243    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 03:02:42.253599 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.253308    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnh6p\""
Oct 30 03:02:42.524794 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.524031    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zwfb4" event={"ID":"160c1f29-a378-43d0-97d8-be1cc643e2c2","Type":"ContainerStarted","Data":"453237e8b03019d636bdc2c4ae43d65499e23704b07a7094e09edd2ed8aae7ce"}
Oct 30 03:02:42.524794 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.524093    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zwfb4" event={"ID":"160c1f29-a378-43d0-97d8-be1cc643e2c2","Type":"ContainerStarted","Data":"29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e"}
Oct 30 03:02:42.526888 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.526822    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v" event={"ID":"29f57388-aede-46d4-9af1-075947d71aa3","Type":"ContainerStarted","Data":"fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f"}
Oct 30 03:02:42.531563 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.531537    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ptqz4" event={"ID":"a35010af-87bc-4a97-81b6-6803ffd77e25","Type":"ContainerStarted","Data":"77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240"}
Oct 30 03:02:42.533747 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.533686    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-klw22" event={"ID":"3e49f31c-2187-47b3-89d9-1e76ec6f883a","Type":"ContainerStarted","Data":"54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830"}
Oct 30 03:02:42.535776 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.535706    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ptzxj" event={"ID":"d1d1d308-d556-4dbc-bd3a-2ec2c838413a","Type":"ContainerStarted","Data":"dfe10a80d99eab09421f80421d8684295aaaa4d43368dcdb5013713d3a8da4a5"}
Oct 30 03:02:42.535776 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.535746    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ptzxj" event={"ID":"d1d1d308-d556-4dbc-bd3a-2ec2c838413a","Type":"ContainerStarted","Data":"58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5"}
Oct 30 03:02:42.538274 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.538215    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk" event={"ID":"1e29e28e-149b-4e87-af1c-a1ab78aa1969","Type":"ContainerStarted","Data":"431f08d175a491f61ad3f8825321b8ee14484333c8cef8d87fcd252d42fa9c6e"}
Oct 30 03:02:42.539908 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.539853    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" event={"ID":"3a65983f-48e3-4e8f-a576-2110dfbb2ec2","Type":"ContainerStarted","Data":"dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd"}
Oct 30 03:02:42.545196 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.544716    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gt4dd" event={"ID":"af46b57a-5155-4ddd-90d8-bf2eb36f8319","Type":"ContainerStarted","Data":"e8052244b785e5b71e123029004e7266415f9a20484086bcad742d5449fcf9cf"}
Oct 30 03:02:42.545196 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.544756    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gt4dd" event={"ID":"af46b57a-5155-4ddd-90d8-bf2eb36f8319","Type":"ContainerStarted","Data":"d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc"}
Oct 30 03:02:42.547051 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.546993    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4" event={"ID":"f6a644f9-7b06-4072-8052-704244dd5a8d","Type":"ContainerStarted","Data":"146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54"}
Oct 30 03:02:42.551247 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.551198    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr" event={"ID":"b7114a74-db33-4c40-a401-030ffa957f21","Type":"ContainerStarted","Data":"3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3"}
Oct 30 03:02:42.560936 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.560729    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zwfb4" podStartSLOduration=433.560712662 podStartE2EDuration="7m13.560712662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:42.55886634 +0000 UTC m=+31.942709814" watchObservedRunningTime="2026-10-30 03:02:42.560712662 +0000 UTC m=+31.944556133"
Oct 30 03:02:42.593121 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.593073    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ptzxj" podStartSLOduration=433.593058206 podStartE2EDuration="7m13.593058206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:42.591666571 +0000 UTC m=+31.975510043" watchObservedRunningTime="2026-10-30 03:02:42.593058206 +0000 UTC m=+31.976901681"
Oct 30 03:02:42.627334 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.627280    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gt4dd" podStartSLOduration=433.627264463 podStartE2EDuration="7m13.627264463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:42.626617531 +0000 UTC m=+32.010461007" watchObservedRunningTime="2026-10-30 03:02:42.627264463 +0000 UTC m=+32.011107938"
Oct 30 03:02:42.698295 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.698254    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:42.699365 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.699342    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:42.699477 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.699370    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:42.700388 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.700354    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:42.700500 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.700443    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:42.700591 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.700575    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:42.700667 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.700627    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.700610891 +0000 UTC m=+34.084454349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:42.701026 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.701009    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.700993638 +0000 UTC m=+34.084837097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:42.701158 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.701144    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:42.701210 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.701190    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.701178284 +0000 UTC m=+34.085021739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:42.801210 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.801133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:42.801359 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.801209    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:42.801359 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.801293    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:42.801359 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.801352    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:42.801546 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:42.801397    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:42.801597 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.801549    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.801531711 +0000 UTC m=+34.185375170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:42.801948 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.801930    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:42.802010 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.801987    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.801972443 +0000 UTC m=+34.185815896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:42.802068 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802051    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:42.802068 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802064    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:42.802170 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802095    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.802085406 +0000 UTC m=+34.185928870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:42.802170 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802147    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:42.802282 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802175    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.802166571 +0000 UTC m=+34.186010029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:42.802282 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802229    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:42.802282 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:42.802258    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:44.802248172 +0000 UTC m=+34.186091630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:44.561882 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.561841    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr" event={"ID":"b7114a74-db33-4c40-a401-030ffa957f21","Type":"ContainerStarted","Data":"ee8b4d4d9db229993f9816d3e8e0dbd41fdbf50b487002dfacfda3b9cf831d5c"}
Oct 30 03:02:44.597357 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.597281    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr" podStartSLOduration=408.385570511 podStartE2EDuration="6m50.597262085s" totalImagesPullingTime="2.211691574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:44.59618799 +0000 UTC m=+33.980031465" watchObservedRunningTime="2026-10-30 03:02:44.597262085 +0000 UTC m=+33.981105560"
Oct 30 03:02:44.727048 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.727011    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:44.727207 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.727075    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:44.727207 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.727121    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:44.727310 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727298    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:44.727358 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727311    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:44.727358 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727355    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.727342742 +0000 UTC m=+38.111186206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:44.728343 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727693    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:44.728343 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727732    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.727722176 +0000 UTC m=+38.111565641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:44.728343 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727776    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:44.728343 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.727795    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.727789603 +0000 UTC m=+38.111633068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:44.827914 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.827835    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:44.827914 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.827882    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:44.827914 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.827917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:44.828190 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.827951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:44.828190 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.827985    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:44.828190 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.828016    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:44.828190 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828067    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.828044755 +0000 UTC m=+38.211888220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:44.828394 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828231    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.828215394 +0000 UTC m=+38.212058848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:44.828394 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828306    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:44.828394 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828339    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.828328596 +0000 UTC m=+38.212172051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:44.828579 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828413    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:44.828579 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828436    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:44.828579 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828449    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:44.828579 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828467    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.828457162 +0000 UTC m=+38.212300617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:44.828579 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.828488    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.828475432 +0000 UTC m=+38.212318884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:44.929221 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.929185    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:44.929411 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.929252    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:44.929890 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.929866    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:02:44.929991 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:44.929954    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:03:16.929935432 +0000 UTC m=+66.313778888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : secret "metrics-daemon-secret" not found
Oct 30 03:02:44.933245 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:44.933220    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:45.029870 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.029814    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:45.034661 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.034630    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:45.159640 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.159607    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:45.160712 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.160681    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:45.481684 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.481659    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rtd54"]
Oct 30 03:02:45.533545 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.533520    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ssh4"]
Oct 30 03:02:45.569839 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.569804    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" event={"ID":"3a65983f-48e3-4e8f-a576-2110dfbb2ec2","Type":"ContainerStarted","Data":"9e6b00fd872a21fdafd7788f2db408438e0139105f779cf78ac7bd0fd82981b9"}
Oct 30 03:02:45.570818 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.570794    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:45.572757 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.572731    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ssh4" event={"ID":"3790bf39-be15-4435-8e27-f59fd78ce1af","Type":"ContainerStarted","Data":"890c119ab4f9a73e34ef13f21e49aa7d2e2871fa8ecdef90bc31d77eb7750343"}
Oct 30 03:02:45.575094 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.575042    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rtd54" event={"ID":"bce76cad-b673-431c-b6a4-40298db25172","Type":"ContainerStarted","Data":"8f15379a8fe5c1289b5899627b8dcd8faab37cfead249863bac919283d21d6ea"}
Oct 30 03:02:45.580770 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.580502    2586 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-qjtcv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Oct 30 03:02:45.580770 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.580548    2586 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused"
Oct 30 03:02:45.608033 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:45.607974    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podStartSLOduration=407.854568373 podStartE2EDuration="6m51.607953212s" totalImagesPullingTime="3.753384839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:45.606842825 +0000 UTC m=+34.990686312" watchObservedRunningTime="2026-10-30 03:02:45.607953212 +0000 UTC m=+34.991796687"
Oct 30 03:02:46.584179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.583787    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/0.log"
Oct 30 03:02:46.584179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.583836    2586 generic.go:356] "Generic (PLEG): container finished" podID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" containerID="9e6b00fd872a21fdafd7788f2db408438e0139105f779cf78ac7bd0fd82981b9" exitCode=255
Oct 30 03:02:46.584179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.583980    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" event={"ID":"3a65983f-48e3-4e8f-a576-2110dfbb2ec2","Type":"ContainerDied","Data":"9e6b00fd872a21fdafd7788f2db408438e0139105f779cf78ac7bd0fd82981b9"}
Oct 30 03:02:46.584179 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.584134    2586 scope.go:122] "RemoveContainer" containerID="9e6b00fd872a21fdafd7788f2db408438e0139105f779cf78ac7bd0fd82981b9"
Oct 30 03:02:46.586991 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.586944    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-klw22" event={"ID":"3e49f31c-2187-47b3-89d9-1e76ec6f883a","Type":"ContainerStarted","Data":"e4cf8d9f84dcd4c245051bfdbec4bd0964b4e14033ce053f96bb86a2e8a4194a"}
Oct 30 03:02:46.661818 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.661760    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-klw22" podStartSLOduration=404.369259654 podStartE2EDuration="6m48.661740389s" totalImagesPullingTime="4.292480735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:46.660456439 +0000 UTC m=+36.044299907" watchObservedRunningTime="2026-10-30 03:02:46.661740389 +0000 UTC m=+36.045583864"
Oct 30 03:02:46.902923 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.901461    2586 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9xq4n"]
Oct 30 03:02:46.923959 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.923931    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:46.937726 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.937701    2586 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9xq4n"]
Oct 30 03:02:46.959072 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.958859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-data-volume\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:46.959072 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.958931    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:46.959072 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.958998    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-crio-socket\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:46.959072 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.959029    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b494s\" (UniqueName: \"kubernetes.io/projected/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-api-access-b494s\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:46.959375 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:46.959271    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.026083 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.026054    2586 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 03:02:47.026233 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.026144    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 03:02:47.026301 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.026242    2586 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qj6bv\""
Oct 30 03:02:47.060240 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b494s\" (UniqueName: \"kubernetes.io/projected/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-api-access-b494s\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.060394 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.060480 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-data-volume\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.060555 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060497    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.060555 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-crio-socket\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.060640 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060618    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-crio-socket\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.060707 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.060690    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:47.060772 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.060757    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.560737177 +0000 UTC m=+36.944580643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:47.060909 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.060890    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-data-volume\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.061015 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.060997    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:47.061079 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.061053    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.561037564 +0000 UTC m=+36.944881031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:47.081153 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.081127    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b494s\" (UniqueName: \"kubernetes.io/projected/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-api-access-b494s\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.570735 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.570703    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.570896 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.570834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:47.570896 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.570876    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:47.570988 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.570933    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:47.570988 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.570953    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.570932495 +0000 UTC m=+37.954775963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:47.570988 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.570976    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.570964202 +0000 UTC m=+37.954807667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:47.593651 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.593586    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk" event={"ID":"1e29e28e-149b-4e87-af1c-a1ab78aa1969","Type":"ContainerStarted","Data":"937c65182e08075fdc82fdab0cdd8f5c7685e6a1db6136e7033db148d0727019"}
Oct 30 03:02:47.595822 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.595802    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/1.log"
Oct 30 03:02:47.596292 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.596272    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/0.log"
Oct 30 03:02:47.596404 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.596306    2586 generic.go:356] "Generic (PLEG): container finished" podID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a" exitCode=255
Oct 30 03:02:47.596404 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.596397    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" event={"ID":"3a65983f-48e3-4e8f-a576-2110dfbb2ec2","Type":"ContainerDied","Data":"086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a"}
Oct 30 03:02:47.596553 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.596537    2586 scope.go:122] "RemoveContainer" containerID="9e6b00fd872a21fdafd7788f2db408438e0139105f779cf78ac7bd0fd82981b9"
Oct 30 03:02:47.596910 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.596879    2586 scope.go:122] "RemoveContainer" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a"
Oct 30 03:02:47.597123 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:47.597089    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qjtcv_openshift-console-operator(3a65983f-48e3-4e8f-a576-2110dfbb2ec2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2"
Oct 30 03:02:47.598950 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.598922    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4" event={"ID":"f6a644f9-7b06-4072-8052-704244dd5a8d","Type":"ContainerStarted","Data":"774e9e0339d9c2fd8620171bfd12176d4cdebbc215cdc9fff7cb7c124421f1cd"}
Oct 30 03:02:47.600812 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.600792    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ptqz4" event={"ID":"a35010af-87bc-4a97-81b6-6803ffd77e25","Type":"ContainerStarted","Data":"0787c3bf8e63c184237957a378091bf0c59812493ff4a4fcf38e47562e827280"}
Oct 30 03:02:47.674745 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.673059    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk" podStartSLOduration=408.226191443 podStartE2EDuration="6m53.67304096s" totalImagesPullingTime="5.446849517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:47.629487961 +0000 UTC m=+37.013331436" watchObservedRunningTime="2026-10-30 03:02:47.67304096 +0000 UTC m=+37.056884437"
Oct 30 03:02:47.708317 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:47.708256    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4" podStartSLOduration=403.893027507 podStartE2EDuration="6m48.70824271s" totalImagesPullingTime="4.815215203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:47.70699584 +0000 UTC m=+37.090839315" watchObservedRunningTime="2026-10-30 03:02:47.70824271 +0000 UTC m=+37.092086184"
Oct 30 03:02:48.582734 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.582408    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:48.582734 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.582550    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:48.582734 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.582587    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:48.582734 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.582627    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.582606076 +0000 UTC m=+39.966449546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:48.582734 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.582642    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:48.582734 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.582684    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.582673071 +0000 UTC m=+39.966516539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:48.607261 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.607240    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/1.log"
Oct 30 03:02:48.607674 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.607643    2586 scope.go:122] "RemoveContainer" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a"
Oct 30 03:02:48.607869 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.607848    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qjtcv_openshift-console-operator(3a65983f-48e3-4e8f-a576-2110dfbb2ec2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2"
Oct 30 03:02:48.609899 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.609348    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v" event={"ID":"29f57388-aede-46d4-9af1-075947d71aa3","Type":"ContainerStarted","Data":"724fcc38863e87a86f074a0208daa19b3843b816cc25a50a62a7325ab3533b8b"}
Oct 30 03:02:48.647259 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.647214    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ptqz4" podStartSLOduration=3.237004016 podStartE2EDuration="8.647202049s" totalImagesPullingTime="5.410198033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:47.747005214 +0000 UTC m=+37.130848692" watchObservedRunningTime="2026-10-30 03:02:48.647202049 +0000 UTC m=+38.031045522"
Oct 30 03:02:48.680925 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.680865    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v" podStartSLOduration=401.466554611 podStartE2EDuration="6m47.680847287s" totalImagesPullingTime="6.214292676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:48.680488523 +0000 UTC m=+38.064331998" watchObservedRunningTime="2026-10-30 03:02:48.680847287 +0000 UTC m=+38.064690763"
Oct 30 03:02:48.784781 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.784739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:48.784949 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.784815    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:48.784949 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.784924    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:48.785076 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.784979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:48.785076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.785004    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.784983609 +0000 UTC m=+46.168827084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:48.785076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.784928    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:48.785076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.785038    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:48.785076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.785048    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:48.785076 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.785079    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.785069059 +0000 UTC m=+46.168912511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:48.785342 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.785168    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.785158838 +0000 UTC m=+46.169002289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:48.886391 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.886310    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:48.886550 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.886406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:48.886550 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.886479    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:48.886550 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886492    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:48.886550 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.886512    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:48.886567    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886515    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886544    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886646    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.88662659 +0000 UTC m=+46.270470055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886668    2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886678    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.886660273 +0000 UTC m=+46.270503732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886578    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886702    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.886691165 +0000 UTC m=+46.270534622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:48.886737 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886718    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.886708565 +0000 UTC m=+46.270552017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:48.887048 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:48.886747    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.886729128 +0000 UTC m=+46.270572598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:49.614941 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:49.614899    2586 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ssh4" event={"ID":"3790bf39-be15-4435-8e27-f59fd78ce1af","Type":"ContainerStarted","Data":"39828904093e27ef0078ec65b3a58bba6139ed987258f6df1b4907d4aeec7052"}
Oct 30 03:02:49.615508 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:49.614992    2586 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:49.654323 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:49.654264    2586 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6ssh4" podStartSLOduration=34.655903096 podStartE2EDuration="37.654246898s" totalImagesPullingTime="2.998343802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:49.653672199 +0000 UTC m=+39.037515673" watchObservedRunningTime="2026-10-30 03:02:49.654246898 +0000 UTC m=+39.038090376"
Oct 30 03:02:50.602980 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:50.602945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:50.603175 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:50.603041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:50.603175 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:50.603073    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:50.603175 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:50.603152    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.603134332 +0000 UTC m=+43.986977786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:50.603371 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:50.603199    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:50.603371 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:50.603251    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.603239655 +0000 UTC m=+43.987083110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:51.324707 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:51.324683    2586 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:51.325022 ip-10-0-135-226 kubenswrapper[2586]: I1030 03:02:51.325008    2586 scope.go:122] "RemoveContainer" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a"
Oct 30 03:02:51.325185 ip-10-0-135-226 kubenswrapper[2586]: E1030 03:02:51.325166    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qjtcv_openshift-console-operator(3a65983f-48e3-4e8f-a576-2110dfbb2ec2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2"
Oct 30 03:02:51.511727 ip-10-0-135-226 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 03:02:51.530913 ip-10-0-135-226 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 03:02:51.531177 ip-10-0-135-226 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 03:02:51.531245 ip-10-0-135-226 systemd[1]: kubelet.service: Consumed 2.925s CPU time.
Oct 30 03:02:51.542112 ip-10-0-135-226 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 03:02:51.610314 ip-10-0-135-226 kubenswrapper[4877]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:02:51.610314 ip-10-0-135-226 kubenswrapper[4877]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 03:02:51.610314 ip-10-0-135-226 kubenswrapper[4877]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:02:51.610314 ip-10-0-135-226 kubenswrapper[4877]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 03:02:51.612361 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612280    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.612361 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612361    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612365    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612369    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612372    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612377    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612381    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612385    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612388    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612391    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612395    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612399    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612403    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612406    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612409    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612411    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612414    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612417    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.612440 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612435    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612450    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612456    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612459    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612463    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612466    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612468    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612471    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612474    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612476    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612479    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612481    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612484    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612487    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612490    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612493    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612496    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612498    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612501    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612503    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.612880 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612506    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612508    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612511    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612514    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612516    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612519    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612522    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612524    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612527    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612530    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612533    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612536    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612539    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612541    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612545    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612548    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612550    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612553    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612556    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612558    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.613397 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612561    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612563    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612566    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612569    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612572    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612574    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612577    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612579    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612582    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612584    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612587    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612590    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612592    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612595    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612598    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612601    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612604    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612606    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612609    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.613919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612611    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612615    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612617    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612620    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612626    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612628    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612631    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612634    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612637    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612639    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612642    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612644    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612647    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612650    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612652    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612655    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612658    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612660    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612663    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612665    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.614401 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612668    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612671    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612673    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612676    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612679    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612683    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612686    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612689    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.612691    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613490    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613500    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613504    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613508    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613512    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613515    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613519    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613524    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613527    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613529    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.614917 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613532    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613535    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613537    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613540    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613542    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613545    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613548    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613550    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613553    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613555    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613558    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613561    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613564    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613567    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613570    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613572    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613575    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613578    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613581    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613584    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.615601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613586    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613603    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613608    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613611    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613614    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613617    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613620    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613623    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613626    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613630    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613634    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613637    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613640    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613643    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613646    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613648    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613651    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613654    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613657    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.616098 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613659    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613663    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613666    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613668    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613671    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613673    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613676    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613679    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613681    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613684    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613686    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613689    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613692    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613695    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613698    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613700    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613703    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613705    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613708    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613710    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.616607 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613713    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613716    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613719    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613721    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613724    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613726    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613729    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613732    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613735    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613737    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613740    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613742    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613745    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613748    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613750    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613753    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613756    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613758    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613761    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613763    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.617170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613766    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613769    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613771    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613774    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613777    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613780    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613782    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613785    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613788    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613790    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613793    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613795    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613798    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613801    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613810    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613814    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.613817    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614307    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614313    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.617688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614317    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614321    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614325    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614329    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614331    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614334    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614337    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614344    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614346    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614349    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614352    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614355    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614357    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614360    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614363    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614365    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614368    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614370    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614374    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.618194 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614378    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614381    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614384    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614387    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614391    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614394    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614396    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614399    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614402    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614405    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614408    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614411    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614414    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614416    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614433    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614437    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614439    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614442    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614444    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614447    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.618685 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614451    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614454    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614457    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614459    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614462    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614465    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614468    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614470    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614473    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614475    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614478    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614481    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614483    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614486    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614488    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614491    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614494    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614497    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614500    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.619180 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614502    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614505    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614508    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614511    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614513    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614516    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614518    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614521    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614524    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614527    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614529    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614531    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614534    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614538    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614541    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614543    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614546    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614548    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614551    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.619677 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614554    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614557    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614560    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614562    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614565    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614567    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614570    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614572    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614575    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614577    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614580    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614583    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614585    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614588    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614590    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614593    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614596    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614599    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614601    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.620148 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614604    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614606    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614609    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614612    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614614    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614618    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614620    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.614624    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614728    4877 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614739    4877 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614750    4877 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614757    4877 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614764    4877 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614769    4877 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614777    4877 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614782    4877 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614785    4877 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614791    4877 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614794    4877 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614798    4877 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614801    4877 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614805    4877 flags.go:64] FLAG: --cgroup-root=""
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614808    4877 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 03:02:51.620629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614811    4877 flags.go:64] FLAG: --client-ca-file=""
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614814    4877 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614817    4877 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614821    4877 flags.go:64] FLAG: --cluster-domain=""
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614824    4877 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614827    4877 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614831    4877 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614834    4877 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614839    4877 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614842    4877 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614846    4877 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614849    4877 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614852    4877 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614855    4877 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614858    4877 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614862    4877 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614866    4877 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614871    4877 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614874    4877 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614877    4877 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614880    4877 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614883    4877 flags.go:64] FLAG: --enable-server="true"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614886    4877 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614891    4877 flags.go:64] FLAG: --event-burst="100"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614894    4877 flags.go:64] FLAG: --event-qps="50"
Oct 30 03:02:51.621207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614897    4877 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614902    4877 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614905    4877 flags.go:64] FLAG: --eviction-hard=""
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614909    4877 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614912    4877 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614916    4877 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614919    4877 flags.go:64] FLAG: --eviction-soft=""
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614922    4877 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614925    4877 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614929    4877 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614932    4877 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614935    4877 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614938    4877 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614941    4877 flags.go:64] FLAG: --feature-gates=""
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614945    4877 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614948    4877 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614952    4877 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614956    4877 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614959    4877 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614962    4877 flags.go:64] FLAG: --help="false"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614965    4877 flags.go:64] FLAG: --hostname-override="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614968    4877 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614972    4877 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 03:02:51.621880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614976    4877 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614979    4877 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614983    4877 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614986    4877 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614990    4877 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614993    4877 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614995    4877 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.614999    4877 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615003    4877 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615006    4877 flags.go:64] FLAG: --kube-reserved=""
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615010    4877 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615014    4877 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615017    4877 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615020    4877 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615023    4877 flags.go:64] FLAG: --lock-file=""
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615026    4877 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615029    4877 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615032    4877 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615038    4877 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615041    4877 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615044    4877 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615048    4877 flags.go:64] FLAG: --logging-format="text"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615051    4877 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615054    4877 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 03:02:51.622472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615057    4877 flags.go:64] FLAG: --manifest-url=""
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615061    4877 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615066    4877 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615070    4877 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615074    4877 flags.go:64] FLAG: --max-pods="110"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615077    4877 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615080    4877 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615084    4877 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615088    4877 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615091    4877 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615094    4877 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615097    4877 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615105    4877 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615108    4877 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615111    4877 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615114    4877 flags.go:64] FLAG: --pod-cidr=""
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615118    4877 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615121    4877 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615125    4877 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615129    4877 flags.go:64] FLAG: --port="10250"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615132    4877 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615135    4877 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027b9566ec751b6e2"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615139    4877 flags.go:64] FLAG: --qos-reserved=""
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615141    4877 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615145    4877 flags.go:64] FLAG: --register-node="true"
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615149    4877 flags.go:64] FLAG: --register-with-taints=""
Oct 30 03:02:51.623340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615153    4877 flags.go:64] FLAG: --registry-burst="10"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615156    4877 flags.go:64] FLAG: --registry-qps="5"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615159    4877 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615162    4877 flags.go:64] FLAG: --reserved-memory=""
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615166    4877 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615170    4877 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615173    4877 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615176    4877 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615179    4877 flags.go:64] FLAG: --runonce="false"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615186    4877 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615190    4877 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615193    4877 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615196    4877 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615199    4877 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615204    4877 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615208    4877 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615211    4877 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615214    4877 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615217    4877 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615220    4877 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615223    4877 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615227    4877 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615230    4877 flags.go:64] FLAG: --system-cgroups=""
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615233    4877 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615240    4877 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 03:02:51.624201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615243    4877 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615247    4877 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615251    4877 flags.go:64] FLAG: --tls-min-version=""
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615254    4877 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615257    4877 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615260    4877 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615263    4877 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615266    4877 flags.go:64] FLAG: --v="2"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615271    4877 flags.go:64] FLAG: --version="false"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615276    4877 flags.go:64] FLAG: --vmodule=""
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615280    4877 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615283    4877 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615440    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615457    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615460    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615463    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615466    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615470    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615474    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615476    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615479    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615484    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.625149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615486    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615489    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615492    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615494    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615497    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615500    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615503    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615505    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615508    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615510    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615516    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615519    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615522    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615525    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615528    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615530    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615533    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615536    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615539    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615541    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.625816 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615544    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615547    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615549    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615552    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615555    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615557    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615560    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615563    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615566    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615569    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615572    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615576    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615579    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615583    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615587    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615591    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615595    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615598    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.626565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615601    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615604    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615606    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615609    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615614    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615617    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615620    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615622    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615625    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615628    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615631    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615633    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615636    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615639    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615641    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615644    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615647    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615649    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615653    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.627028 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615655    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615658    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615661    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615663    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615667    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615669    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615673    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615676    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615678    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615681    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615683    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615686    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615688    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615691    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615693    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615696    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615699    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615702    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615705    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615708    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.627561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615710    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615713    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615716    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615719    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615721    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615724    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615726    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615729    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615732    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615734    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615737    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615740    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615742    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615745    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615748    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615750    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615753    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615756    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.628168 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615758    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.615788    4877 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615825    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615830    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615833    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615836    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615839    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615841    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615844    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615847    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615849    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615853    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615856    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615859    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615861    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615865    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.628679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615867    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615870    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615873    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615875    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615878    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615880    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615883    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615886    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615888    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615891    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615894    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615896    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615899    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615901    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615904    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615906    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615910    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615912    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615915    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615918    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.629332 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615920    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615923    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615925    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615928    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615930    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615933    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615937    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615941    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615945    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615949    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615953    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615956    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615958    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615961    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615964    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615966    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615969    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615971    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.630192 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615974    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615976    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615979    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615982    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615984    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615987    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615989    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615992    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615994    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615997    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.615999    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616003    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616006    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616009    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616011    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616014    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616016    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616019    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616022    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616024    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.630961 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616026    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616032    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616034    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616037    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616040    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616042    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616045    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616047    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616050    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616053    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616055    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616058    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616060    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616064    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616067    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616069    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616072    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616074    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616077    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616080    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.631857 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616083    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616086    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616088    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616091    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616094    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616096    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616099    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616102    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616104    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616107    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616110    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616113    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616116    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.616121    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.619714    4877 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.619728    4877 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.619747    4877 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.619752    4877 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619808    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619813    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.632727 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619817    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619820    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619823    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619825    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619828    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619830    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619833    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619836    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619838    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619842    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619845    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619847    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619850    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619853    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619855    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619859    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619864    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619867    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619870    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.633811 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619873    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619876    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619879    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619882    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619885    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619888    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619890    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619893    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619896    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619899    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619901    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619904    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619907    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619909    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619912    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619915    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619917    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619920    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619923    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.634658 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619926    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619928    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619931    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619934    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619936    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619939    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619942    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619944    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619947    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619949    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619954    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619959    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619962    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619965    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619968    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619970    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619973    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619976    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619978    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.635489 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619981    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619983    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619986    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619988    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619991    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619994    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619996    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.619999    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620001    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620004    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620007    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620010    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620013    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620015    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620018    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620020    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620023    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620025    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620028    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620031    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.636299 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620033    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620036    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620038    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620041    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620044    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620047    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620049    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620052    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620054    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620057    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620059    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620062    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620064    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620067    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620070    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620072    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620075    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620078    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620080    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620083    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.637235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620086    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620088    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620091    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620094    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620097    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620099    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620101    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.620127    4877 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620266    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620271    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620274    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620276    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620279    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620282    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620285    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620287    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.638222 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620290    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620292    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620296    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620298    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620302    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620306    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620310    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620313    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620316    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620318    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620321    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620324    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620326    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620329    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620331    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620334    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620336    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620339    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620342    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620344    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.638973 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620347    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620349    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620352    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620354    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620357    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620359    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620362    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620364    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620367    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620369    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620372    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620375    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620377    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620379    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620382    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620385    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620388    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620390    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620393    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.639892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620395    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620398    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620400    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620403    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620405    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620407    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620410    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620413    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620415    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620417    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620440    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620443    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620446    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620449    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620451    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620454    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620457    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620459    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620462    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.640779 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620464    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620468    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620472    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620475    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620478    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620485    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620488    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620491    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620493    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620496    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620500    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620503    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620505    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620508    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620511    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620513    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620516    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620518    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620521    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.641642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620524    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620526    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620529    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620531    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620534    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620537    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620539    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620541    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620544    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620547    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620550    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620552    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620555    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620557    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620560    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620563    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620565    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620568    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620572    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620575    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.642457 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620578    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.620601    4877 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620634    4877 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620639    4877 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620643    4877 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620645    4877 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620648    4877 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620651    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620653    4877 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620656    4877 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620659    4877 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620661    4877 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620664    4877 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620666    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620669    4877 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620672    4877 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 03:02:51.643468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620675    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620677    4877 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620680    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620682    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620686    4877 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620689    4877 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620692    4877 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620694    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620697    4877 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620700    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620703    4877 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620705    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620708    4877 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620710    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620713    4877 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620717    4877 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620719    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620722    4877 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620725    4877 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 03:02:51.644088 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620727    4877 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620730    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620733    4877 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620736    4877 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620738    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620741    4877 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620744    4877 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620746    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620749    4877 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620751    4877 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620754    4877 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620756    4877 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620759    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620762    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620764    4877 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620767    4877 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620769    4877 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620772    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620774    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620777    4877 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 03:02:51.644946 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620779    4877 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620782    4877 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620785    4877 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620787    4877 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620791    4877 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620793    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620796    4877 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620798    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620803    4877 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620805    4877 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620808    4877 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620811    4877 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620814    4877 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620816    4877 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620819    4877 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620822    4877 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620826    4877 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620829    4877 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620832    4877 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620836    4877 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 03:02:51.645588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620838    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620842    4877 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620845    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620847    4877 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620850    4877 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620852    4877 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620855    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620857    4877 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620860    4877 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620862    4877 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620865    4877 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620868    4877 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620870    4877 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620873    4877 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620875    4877 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620879    4877 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620891    4877 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620894    4877 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620897    4877 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620900    4877 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 03:02:51.646400 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620904    4877 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620907    4877 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620910    4877 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620912    4877 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620915    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620918    4877 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620920    4877 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620923    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620926    4877 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620929    4877 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620932    4877 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620935    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:51.620938    4877 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.620986    4877 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.621894    4877 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.621942    4877 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.622107    4877 server.go:1014] "Starting client certificate rotation"
Oct 30 03:02:51.647029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.622246    4877 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.622298    4877 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:57:11 +0000 UTC" deadline="2028-04-14 03:50:34.280261344 +0000 UTC"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.622326    4877 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12768h47m42.657938177s"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.622547    4877 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.623736    4877 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.625881    4877 log.go:25] "Validated CRI v1 runtime API"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.627600    4877 log.go:25] "Validated CRI v1 image API"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.628325    4877 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 03:02:51.647609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.630908    4877 fs.go:135] Filesystem UUIDs: map[2c131d26-2f91-4c0c-aa60-f90299cd035d:/dev/nvme0n1p4 6c3df54c-3031-467d-b89f-5cc08c247aeb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 03:02:51.647841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.630938    4877 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41726340b5ec7f10590aab25a069173160ebecedd130e921538b865946ee2081/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41726340b5ec7f10590aab25a069173160ebecedd130e921538b865946ee2081/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/431f08d175a491f61ad3f8825321b8ee14484333c8cef8d87fcd252d42fa9c6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/431f08d175a491f61ad3f8825321b8ee14484333c8cef8d87fcd252d42fa9c6e/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/890c119ab4f9a73e34ef13f21e49aa7d2e2871fa8ecdef90bc31d77eb7750343/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/890c119ab4f9a73e34ef13f21e49aa7d2e2871fa8ecdef90bc31d77eb7750343/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f15379a8fe5c1289b5899627b8dcd8faab37cfead249863bac919283d21d6ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f15379a8fe5c1289b5899627b8dcd8faab37cfead249863bac919283d21d6ea/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/060b1735-396e-43d2-940d-e5b9bd3ec06b/volumes/kubernetes.io~projected/kube-api-access-49rvh:{mountpoint:/var/lib/kubelet/pods/060b1735-396e-43d2-940d-e5b9bd3ec06b/volumes/kubernetes.io~projected/kube-api-access-49rvh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~projected/kube-api-access-km82f:{mountpoint:/var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~projected/kube-api-access-km82f major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~secret/stats-auth major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/160c1f29-a378-43d0-97d8-be1cc643e2c2/volumes/kubernetes.io~projected/kube-api-access-bqq7j:{mountpoint:/var/lib/kubelet/pods/160c1f29-a378-43d0-97d8-be1cc643e2c2/volumes/kubernetes.io~projected/kube-api-access-bqq7j major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e29e28e-149b-4e87-af1c-a1ab78aa1969/volumes/kubernetes.io~projected/kube-api-access-p2hps:{mountpoint:/var/lib/kubelet/pods/1e29e28e-149b-4e87-af1c-a1ab78aa1969/volumes/kubernetes.io~projected/kube-api-access-p2hps major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e29e28e-149b-4e87-af1c-a1ab78aa1969/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1e29e28e-149b-4e87-af1c-a1ab78aa1969/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29f57388-aede-46d4-9af1-075947d71aa3/volumes/kubernetes.io~projected/kube-api-access-9k2mp:{mountpoint:/var/lib/kubelet/pods/29f57388-aede-46d4-9af1-075947d71aa3/volumes/kubernetes.io~projected/kube-api-access-9k2mp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~projected/kube-api-access-zbdb8:{mountpoint:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~projected/kube-api-access-zbdb8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3790bf39-be15-4435-8e27-f59fd78ce1af/volumes/kubernetes.io~projected/kube-api-access-xssbw:{mountpoint:/var/lib/kubelet/pods/3790bf39-be15-4435-8e27-f59fd78ce1af/volumes/kubernetes.io~projected/kube-api-access-xssbw major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a65983f-48e3-4e8f-a576-2110dfbb2ec2/volumes/kubernetes.io~projected/kube-api-access-8549h:{mountpoint:/var/lib/kubelet/pods/3a65983f-48e3-4e8f-a576-2110dfbb2ec2/volumes/kubernetes.io~projected/kube-api-access-8549h major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a65983f-48e3-4e8f-a576-2110dfbb2ec2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3a65983f-48e3-4e8f-a576-2110dfbb2ec2/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e49f31c-2187-47b3-89d9-1e76ec6f883a/volumes/kubernetes.io~projected/kube-api-access-w4bbg:{mountpoint:/var/lib/kubelet/pods/3e49f31c-2187-47b3-89d9-1e76ec6f883a/volumes/kubernetes.io~projected/kube-api-access-w4bbg major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e49f31c-2187-47b3-89d9-1e76ec6f883a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3e49f31c-2187-47b3-89d9-1e76ec6f883a/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volumes/kubernetes.io~projected/kube-api-access-rv4lv:{mountpoint:/var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volumes/kubernetes.io~projected/kube-api-access-rv4lv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66f7168c-a3bc-43e5-a198-b3551b33b770/volumes/kubernetes.io~projected/kube-api-access-fjjfw:{mountpoint:/var/lib/kubelet/pods/66f7168c-a3bc-43e5-a198-b3551b33b770/volumes/kubernetes.io~projected/kube-api-access-fjjfw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74e50c74-23f8-46b7-98e6-6ac103daba15/volumes/kubernetes.io~projected/kube-api-access-nplzq:{mountpoint:/var/lib/kubelet/pods/74e50c74-23f8-46b7-98e6-6ac103daba15/volumes/kubernetes.io~projected/kube-api-access-nplzq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~projected/kube-api-access-sh4bs:{mountpoint:/var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~projected/kube-api-access-sh4bs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/940bd241-206c-44c1-9c91-b6eb875bce7c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/940bd241-206c-44c1-9c91-b6eb875bce7c/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a767b5d-c7e1-4178-8793-4bd9f4352745/volumes/kubernetes.io~projected/kube-api-access-x2vqp:{mountpoint:/var/lib/kubelet/pods/9a767b5d-c7e1-4178-8793-4bd9f4352745/volumes/kubernetes.io~projected/kube-api-access-x2vqp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9/volumes/kubernetes.io~projected/kube-api-access-8krn5:{mountpoint:/var/lib/kubelet/pods/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9/volumes/kubernetes.io~projected/kube-api-access-8krn5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9da977d0-2885-4889-a18f-9038e7e11272/volumes/kubernetes.io~projected/kube-api-access-8xtjx:{mountpoint:/var/lib/kubelet/pods/9da977d0-2885-4889-a18f-9038e7e11272/volumes/kubernetes.io~projected/kube-api-access-8xtjx major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9de66365-30cc-4adc-ad1b-4a6778246af8/volumes/kubernetes.io~projected/kube-api-access-5cj5d:{mountpoint:/var/lib/kubelet/pods/9de66365-30cc-4adc-ad1b-4a6778246af8/volumes/kubernetes.io~projected/kube-api-access-5cj5d major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a35010af-87bc-4a97-81b6-6803ffd77e25/volumes/kubernetes.io~projected/kube-api-access-fz7ct:{mountpoint:/var/lib/kubelet/pods/a35010af-87bc-4a97-81b6-6803ffd77e25/volumes/kubernetes.io~projected/kube-api-access-fz7ct major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a35010af-87bc-4a97-81b6-6803ffd77e25/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a35010af-87bc-4a97-81b6-6803ffd77e25/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e/volumes/kubernetes.io~projected/kube-api-access-b494s:{mountpoint:/var/lib/kubelet/pods/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e/volumes/kubernetes.io~projected/kube-api-access-b494s major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af46b57a-5155-4ddd-90d8-bf2eb36f8319/volumes/kubernetes.io~projected/kube-api-access-8z5dm:{mountpoint:/var/lib/kubelet/pods/af46b57a-5155-4ddd-90d8-bf2eb36f8319/volumes/kubernetes.io~projected/kube-api-access-8z5dm major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7114a74-db33-4c40-a401-030ffa957f21/volumes/kubernetes.io~projected/kube-api-access-k5pzb:{mountpoint:/var/lib/kubelet/pods/b7114a74-db33-4c40-a401-030ffa957f21/volumes/kubernetes.io~projected/kube-api-access-k5pzb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bce76cad-b673-431c-b6a4-40298db25172/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bce76cad-b673-431c-b6a4-40298db25172/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bec84fda-07e9-469d-bec4-16e9586fed14/volumes/kubernetes.io~projected/kube-api-access-kjrj2:{mountpoint:/var/lib/kubelet/pods/bec84fda-07e9-469d-bec4-16e9586fed14/volumes/kubernetes.io~projected/kube-api-access-kjrj2 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1d1d308-d556-4dbc-bd3a-2ec2c838413a/volumes/kubernetes.io~projected/kube-api-access-22fcp:{mountpoint:/var/lib/kubelet/pods/d1d1d308-d556-4dbc-bd3a-2ec2c838413a/volumes/kubernetes.io~projected/kube-api-access-22fcp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e00a6bb6-263b-40c7-977d-11511832d40a/volumes/kubernetes.io~projected/kube-api-access-jd8sr:{mountpoint:/var/lib/kubelet/pods/e00a6bb6-263b-40c7-977d-11511832d40a/volumes/kubernetes.io~projected/kube-api-access-jd8sr major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a644f9-7b06-4072-8052-704244dd5a8d/volumes/kubernetes.io~projected/kube-api-access-qbmg2:{mountpoint:/var/lib/kubelet/pods/f6a644f9-7b06-4072-8052-704244dd5a8d/volumes/kubernetes.io~projected/kube-api-access-qbmg2 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6a644f9-7b06-4072-8052-704244dd5a8d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f6a644f9-7b06-4072-8052-704244dd5a8d/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~projected/kube-api-access-mdsfn:{mountpoint:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~projected/kube-api-access-mdsfn major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdce563b-9436-449d-8cd6-159e7da40143/volumes/kubernetes.io~projected/kube-api-access-zjkx9:{mountpoint:/var/lib/kubelet/pods/fdce563b-9436-449d-8cd6-159e7da40143/volumes/kubernetes.io~projected/kube-api-access-zjkx9 major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/011b0b4187d44dbf0d0c99891a0cf48a5a8016a859af5b1523a3892d25a1604e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9c07d41eb58bbdb56ebff471c6da3423f160826de346f71022d0491d4fa058f0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6079e14ed3ecca3a9bd69363fc3c632438cd61929ad7fcf850cef6f3a5c0761c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/962081d6f3458b75e9c0b495fd69f10d8dd4023a18436dfd3ea6defb09b5c85d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/31731b9b1dd0511fb41ef68113bc0abbe8551395e5b6e801bdadc2d5b1763be1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/017e2ba937f1f7ab5198374d9189e9dee5179dca220d35585aaa6b08122c7d5e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/14991046d89e51671a6fce9e23cc6cbe72b9a184accb8296357aa56d8f14036e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9f79ab8318ad764c45f5cd6e92f9b8b24d7a7017d49c3c0f2fca7f386dbd52e1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/09f773ed6223254352e33fbf01c09420ef02087869dfe93e8852ec3f34568e48/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a1698b981a8b1cf39c5311a28aa67c2bfacd8828effd94e3756de2ad4ff4ff1a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/d3e4625b19ae33afa84b55ee01bab10a9551a04dbc2f2bfc21e03b27577a1d50/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/43d18180c54bdfc41677900fc3f4729e3232a9770f90997ddb6d7a3297b73373/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/3018d77de4dbf1f2d4663b00273295df53752d0997d5bb139449e25a03a3ef8e/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/354a963d13279abfa13ef7b365aac58aa001b0313809631625a733fd34ea76d0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5d73a1e791b30ceebeb83cc6449c805ded764fa25efc6a7a42bb1d1e09085c58/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/29d22423f68303d574b72924b38f795efaefbd4f91eed35be4b3f441821cf467/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4ab81d00eaa8557457b2324141037fc1a9f34b1262a8924e522a11f664510629/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7d5b1fc481917281e86321727a0eb823f0dce5042fda3f94e528ec10d585b8b0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8157a56f594200093f877bd419c058029297de17bb6b0147cf53af52778d2c70/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/eb4e5eef4647bbbbfaf8dabe475b8cd75a2042bd5abd56a611c2886cfa8a71cb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/916ccd398c70a1b52b5f9ed7e1a09fedc262b3bfc17b19757912961e36a77a0a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/cf86189cad35f4ff2687d466542cf625d13f2deea03079303261717da0e06045/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6b44e1d30367b2cd4538ccf8f41e8f777976e797649ead8039009c11025d65db/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/06db77c9d02d14e4626d953f510972706c5737e09daeb205cc8a39179f1a871f/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/ce99c4a3b6beabe0d6a6259951932da64c132c133d4b8a7556a5e66c0f4ab978/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d9f2b6be70789b52ae6f33bf0eeefb0d3aa914b2b2f65b383172686b3f2614c7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/74d4b825c369521f4e96bff4e1912232b9a061bbd8dc85cf69d3a913d88ec0b2/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/21d0078f7dacab3b4c2b4effcddd0fd9e2c9f042a8e7e2f465d157e58dcb23f3/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d1c982af3f43784bed2c2a485fb8ce58d30299e7c643f77eb5ea628c89230393/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/abe8fb6f8a6b225a4daabcd09d841588c4d65fd6cce94e22bdfbfc4f4d95729e/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/63d98491e532cd811aed4e8154f2ff5de4b7385654013777d19b3000c1e0b478/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/1fb0d1133aa6de9f926c360c413a0d749fc6f19079361464dd508399d116c242/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/b555c999ef5a110ac33614df4eedfc2fa23904415024e674dc31661ddbeff09d/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/af2c0f9a5d918f57edf69046cb3f1b77958ce5c5b7696c13d4fdebdaef232002/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/2ad704879201f0dce998007cd5c485f8c90a3c147cd6e49dad09d0f12be209e3/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/54c4d600d6c008ccf4f9d3ef803cdd8108689baabea9e48fe220ef99456519ee/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/3d06adf027f35818960bdfc3a6ec5f8d5d13dfc4f2e0e0227940e1e371e41bbc/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/3a569bd1ae5358bbe1c338ed9d96cfab864b827e19fbbdc1c1f20dabc9dcb70f/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/647f8df827a410bf5dbbd89d69ab95a2b6a539f54c92a0a4c5af4f6b2f536686/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/4cba4e6d9f745db13c69a1d0115a72712cf7162b5f418c7b37363228959bc631/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/33aea53f55bad9964ea6099ffa3a6e80f248b487163fd50f8be2c8db098cce90/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/5f5615d39f1a395cc9d2f54ef4358ee16d5efe37dc525f512005eebd49fd2ae6/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/097079b7d529acd0d9a3e09d833fd266c61abefc42f4614a120e55d5b2768545/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/5cf838b19e0626f1960dab5a1286d8dafd7ad3908b1dfe733a95da2d221b70cd/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/055345fd663efd5b39fd8c28ff640e5f24f149f4dc6727d7f09ab1d3f00539da/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7aacafce96cee3bb1ca69c2e7170953d39ecdb2e342a665581792938b2e92e1a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/864c3268237c72ac424fb54c2f1af95d01ecad560f061bd2840652754b865696/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bf9761da6cb682a945a42584f8285c80c570995dd69ebfda9218205d1e087ed0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/20bd09a380be1504387646348b7807eed9876acb1c5afacb71b7d53f3a643ea4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6a80b3a9fe27912038cb4546ec5deff3ff5b27324626b20557d4ee6737d3fa75/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d652dfc009fdeeb835927c85ce7882981e60adc16ce9f9a58c71a982249374b2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ee807be1aba1108bf11eaa54cfc68abfdeab5955752919c4357c877a2f51078d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1f14786bc1c4176abd50673ee96b6900708ca4523c7696c6f79d24e49c830212/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ad38c75c9b83a545175fdcffe643d1bfce50c62183493ea9d0b69d23c591836f/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.642024    4877 manager.go:217] Machine: {Timestamp:2026-10-30 03:02:51.641218681 +0000 UTC m=+0.070108094 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122549 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e38677ed165f104fe4b7b80cd85eb SystemUUID:ec2e3867-7ed1-65f1-04fe-4b7b80cd85eb BootID:ef77d2a0-d2bf-434d-9df5-e7d8324b997f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66f7168c-a3bc-43e5-a198-b3551b33b770/volumes/kubernetes.io~projected/kube-api-access-fjjfw DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9/volumes/kubernetes.io~projected/kube-api-access-8krn5 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e7b33e85ead4deaa6803e4b482b238a77db1e787c09bc2a8617bfd5fc82871a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/af46b57a-5155-4ddd-90d8-bf2eb36f8319/volumes/kubernetes.io~projected/kube-api-access-8z5dm DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~projected/kube-api-access-mdsfn DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3790bf39-be15-4435-8e27-f59fd78ce1af/volumes/kubernetes.io~projected/kube-api-access-xssbw DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e/volumes/kubernetes.io~projected/kube-api-access-b494s DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e00a6bb6-263b-40c7-977d-11511832d40a/volumes/kubernetes.io~projected/kube-api-access-jd8sr DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d1d1d308-d556-4dbc-bd3a-2ec2c838413a/volumes/kubernetes.io~projected/kube-api-access-22fcp DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54b37426c4efd117ee1359a6626c9c021bcf2776acd8792520e3e9a2dd462830/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77451cfe0e9b83dd6dda71ae8e6fc31f6d9ec3d2e2eea338b2be2f1216add240/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/940bd241-206c-44c1-9c91-b6eb875bce7c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b5cbd8dd1f34acb033e7d6b94a70be867d53211f26eddc6c286d80afd241a18/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e29e28e-149b-4e87-af1c-a1ab78aa1969/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a35010af-87bc-4a97-81b6-6803ffd77e25/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41ac6f4ffaa306cad7d817fdc8b508eae00ad804ef769a75e2aeff0abe0b037f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a767b5d-c7e1-4178-8793-4bd9f4352745/volumes/kubernetes.io~projected/kube-api-access-x2vqp DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e49f31c-2187-47b3-89d9-1e76ec6f883a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29f57388-aede-46d4-9af1-075947d71aa3/volumes/kubernetes.io~projected/kube-api-access-9k2mp DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3df380284b5477dfb70ab1344b1a68d3341f1b3aaa327237618503294d7b54c3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/890c119ab4f9a73e34ef13f21e49aa7d2e2871fa8ecdef90bc31d77eb7750343/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9da977d0-2885-4889-a18f-9038e7e11272/volumes/kubernetes.io~projected/kube-api-access-8xtjx DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd8e6eb39c55a8838192119c96b13c160cca673389982da7128174c95d082edd/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/060b1735-396e-43d2-940d-e5b9bd3ec06b/volumes/kubernetes.io~projected/kube-api-access-49rvh DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a65983f-48e3-4e8f-a576-2110dfbb2ec2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/160c1f29-a378-43d0-97d8-be1cc643e2c2/volumes/kubernetes.io~projected/kube-api-access-bqq7j DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29088aac7d358dfc736b8035b8bb75338df4bcee4145a95dffadff2c67fe041e/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9de66365-30cc-4adc-ad1b-4a6778246af8/volumes/kubernetes.io~projected/kube-api-access-5cj5d DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~projected/kube-api-access-zbdb8 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63498c86-702e-47d4-8002-0693eb34b05e/volumes/kubernetes.io~projected/kube-api-access-rv4lv DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41726340b5ec7f10590aab25a069173160ebecedd130e921538b865946ee2081/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bec84fda-07e9-469d-bec4-16e9586fed14/volumes/kubernetes.io~projected/kube-api-access-kjrj2 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74e50c74-23f8-46b7-98e6-6ac103daba15/volumes/kubernetes.io~projected/kube-api-access-nplzq DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a644f9-7b06-4072-8052-704244dd5a8d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f15379a8fe5c1289b5899627b8dcd8faab37cfead249863bac919283d21d6ea/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8046eaad-8e51-49d2-beac-8f6be5170f4d/volumes/kubernetes.io~projected/kube-api-access-sh4bs DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e49f31c-2187-47b3-89d9-1e76ec6f883a/volumes/kubernetes.io~projected/kube-api-access-w4bbg DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1569b662-064a-4998-b332-03c6ed48e312/volumes/kubernetes.io~projected/kube-api-access-km82f DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd5fc1c6763275656a67d84194f89209168a61f476ba85095db54195a5f5da2b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7114a74-db33-4c40-a401-030ffa957f21/volumes/kubernetes.io~projected/kube-api-access-k5pzb DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f6a644f9-7b06-4072-8052-704244dd5a8d/volumes/kubernetes.io~projected/kube-api-access-qbmg2 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58d46604aa9f4e7b2c95f9986f25359aa2444c628e1fffcf22411fc0879755d5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d24865cd0ba3afcadbb7bbf2fe5f34449530f8cfd21e139e309469b990948ccc/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/146e8bac9cba1b1ab5dc01913192200ecd21c75febc3694c6696a5b549e92e54/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdce563b-9436-449d-8cd6-159e7da40143/volumes/kubernetes.io~projected/kube-api-access-zjkx9 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaf73b9010deb95f480a159568c416307f73a480cc5f2072c98e6741590c2cc5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e29e28e-149b-4e87-af1c-a1ab78aa1969/volumes/kubernetes.io~projected/kube-api-access-p2hps DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a35010af-87bc-4a97-81b6-6803ffd77e25/volumes/kubernetes.io~projected/kube-api-access-fz7ct DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/431f08d175a491f61ad3f8825321b8ee14484333c8cef8d87fcd252d42fa9c6e/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c25c3948ca1f93d0c4039f23c428cc153ff5677338d8fa1aab443b44dab44e1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a65983f-48e3-4e8f-a576-2110dfbb2ec2/volumes/kubernetes.io~projected/kube-api-access-8549h DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bce76cad-b673-431c-b6a4-40298db25172/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fef10b6f0944c6b2384bf9aa5d5c0e4bcdc18f350cb9a09e2d4e2a23da5b661f/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6725e6e81e42c02943b254a7aff84b658293f411c6be4c1359f67e2cb5332c8e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:146e8bac9cba1b1 MacAddress:62:66:56:54:86:31 Speed:10000 Mtu:8901} {Name:29088aac7d358df MacAddress:5e:ef:e7:ce:5e:6b Speed:10000 Mtu:8901} {Name:3df380284b5477d MacAddress:16:59:fc:e5:33:63 Speed:10000 Mtu:8901} {Name:431f08d175a491f MacAddress:6e:d3:00:2b:b5:da Speed:10000 Mtu:8901} {Name:54b37426c4efd11 MacAddress:52:d3:ca:f3:cf:86 Speed:10000 Mtu:8901} {Name:58d46604aa9f4e7 MacAddress:02:67:c7:04:4c:e0 Speed:10000 Mtu:8901} {Name:77451cfe0e9b83d MacAddress:f2:9a:5d:db:67:d5 Speed:10000 Mtu:8901} {Name:890c119ab4f9a73 MacAddress:62:31:ed:d7:f8:e2 Speed:10000 Mtu:8901} {Name:8f15379a8fe5c12 MacAddress:42:9a:ca:cd:21:86 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f5:9f:70:11 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:ef:d1:e5:28:86 Speed:0 Mtu:8901} {Name:d24865cd0ba3afc MacAddress:26:68:9b:f0:3e:32 Speed:10000 Mtu:8901} {Name:dd8e6eb39c55a88 MacAddress:56:25:7a:b7:8c:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f5:9f:70:11 Speed:0 Mtu:9001} {Name:fef10b6f0944c6b MacAddress:82:07:a5:11:c4:cd Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:82:1b:7c:36:c0:e2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:8c:dd:99:7b:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.642590    4877 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.642647    4877 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.642962    4877 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.642990    4877 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643213    4877 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643226    4877 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643243    4877 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643262    4877 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643287    4877 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643512    4877 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643669    4877 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643712    4877 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643725    4877 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643739    4877 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643753    4877 kubelet.go:405] "Adding apiserver pod source"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.643763    4877 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.644674    4877 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.644687    4877 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:02:51.648039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.645177    4877 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 03:02:51.649243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649226    4877 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 03:02:51.649243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649243    4877 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 03:02:51.649347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649337    4877 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 03:02:51.649546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649534    4877 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 03:02:51.649578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649564    4877 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 03:02:51.649663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649654    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 03:02:51.649695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649668    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 03:02:51.649695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649674    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 03:02:51.649695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649679    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 03:02:51.649695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649685    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 03:02:51.649695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649690    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 03:02:51.649695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649696    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649705    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649713    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649718    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649726    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649735    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649761    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 03:02:51.649858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.649770    4877 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 03:02:51.653369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.653356    4877 server.go:1263] "Started kubelet"
Oct 30 03:02:51.653461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.653405    4877 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 03:02:51.653510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.653460    4877 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 03:02:51.653565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.653528    4877 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 03:02:51.653898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.653801    4877 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 03:02:51.653906 ip-10-0-135-226 systemd[1]: Started Kubernetes Kubelet.
Oct 30 03:02:51.654798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.654746    4877 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 03:02:51.656623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.656603    4877 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 03:02:51.657157 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.657110    4877 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 03:02:51.658186 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:51.658169    4877 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-226.ec2.internal\" not found"
Oct 30 03:02:51.658331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.658209    4877 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 03:02:51.658331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.658217    4877 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 03:02:51.658526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.658412    4877 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 03:02:51.659857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.659811    4877 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:13 +0000 UTC" deadline="2028-07-23 08:11:56.347157004 +0000 UTC"
Oct 30 03:02:51.659951 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.659858    4877 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15173h9m4.687303628s"
Oct 30 03:02:51.661283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661243    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49f31c-2187-47b3-89d9-1e76ec6f883a" volumeName="kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-tmp" seLinuxMountContext=""
Oct 30 03:02:51.661403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661380    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63498c86-702e-47d4-8002-0693eb34b05e" volumeName="kubernetes.io/secret/63498c86-702e-47d4-8002-0693eb34b05e-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 03:02:51.661535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661519    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a767b5d-c7e1-4178-8793-4bd9f4352745" volumeName="kubernetes.io/configmap/9a767b5d-c7e1-4178-8793-4bd9f4352745-iptables-alerter-script" seLinuxMountContext=""
Oct 30 03:02:51.661617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661605    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bce76cad-b673-431c-b6a4-40298db25172" volumeName="kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret" seLinuxMountContext=""
Oct 30 03:02:51.661686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661676    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec84fda-07e9-469d-bec4-16e9586fed14" volumeName="kubernetes.io/projected/bec84fda-07e9-469d-bec4-16e9586fed14-kube-api-access-kjrj2" seLinuxMountContext=""
Oct 30 03:02:51.661765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661751    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1569b662-064a-4998-b332-03c6ed48e312" volumeName="kubernetes.io/projected/1569b662-064a-4998-b332-03c6ed48e312-kube-api-access-km82f" seLinuxMountContext=""
Oct 30 03:02:51.661840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661820    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e29e28e-149b-4e87-af1c-a1ab78aa1969" volumeName="kubernetes.io/configmap/1e29e28e-149b-4e87-af1c-a1ab78aa1969-config" seLinuxMountContext=""
Oct 30 03:02:51.661840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661839    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49f31c-2187-47b3-89d9-1e76ec6f883a" volumeName="kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-service-ca-bundle" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661866    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8046eaad-8e51-49d2-beac-8f6be5170f4d" volumeName="kubernetes.io/projected/8046eaad-8e51-49d2-beac-8f6be5170f4d-kube-api-access-sh4bs" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661878    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a644f9-7b06-4072-8052-704244dd5a8d" volumeName="kubernetes.io/configmap/f6a644f9-7b06-4072-8052-704244dd5a8d-config" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661894    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661922    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661935    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.661958    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" volumeName="kubernetes.io/projected/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-kube-api-access-8549h" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662001    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63498c86-702e-47d4-8002-0693eb34b05e" volumeName="kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-env-overrides" seLinuxMountContext=""
Oct 30 03:02:51.662012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662013    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662038    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1569b662-064a-4998-b332-03c6ed48e312" volumeName="kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-stats-auth" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662060    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e29e28e-149b-4e87-af1c-a1ab78aa1969" volumeName="kubernetes.io/projected/1e29e28e-149b-4e87-af1c-a1ab78aa1969-kube-api-access-p2hps" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662071    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e00a6bb6-263b-40c7-977d-11511832d40a" volumeName="kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-cni-binary-copy" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662082    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a644f9-7b06-4072-8052-704244dd5a8d" volumeName="kubernetes.io/secret/f6a644f9-7b06-4072-8052-704244dd5a8d-serving-cert" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662102    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74e50c74-23f8-46b7-98e6-6ac103daba15" volumeName="kubernetes.io/projected/74e50c74-23f8-46b7-98e6-6ac103daba15-kube-api-access-nplzq" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662133    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="940bd241-206c-44c1-9c91-b6eb875bce7c" volumeName="kubernetes.io/secret/940bd241-206c-44c1-9c91-b6eb875bce7c-agent-certs" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662169    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="160c1f29-a378-43d0-97d8-be1cc643e2c2" volumeName="kubernetes.io/projected/160c1f29-a378-43d0-97d8-be1cc643e2c2-kube-api-access-bqq7j" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662175    4877 factory.go:55] Registering systemd factory
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662185    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9da977d0-2885-4889-a18f-9038e7e11272" volumeName="kubernetes.io/projected/9da977d0-2885-4889-a18f-9038e7e11272-kube-api-access-8xtjx" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662193    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdce563b-9436-449d-8cd6-159e7da40143" volumeName="kubernetes.io/projected/fdce563b-9436-449d-8cd6-159e7da40143-kube-api-access-zjkx9" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662201    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1569b662-064a-4998-b332-03c6ed48e312" volumeName="kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-default-certificate" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662216    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49f31c-2187-47b3-89d9-1e76ec6f883a" volumeName="kubernetes.io/projected/3e49f31c-2187-47b3-89d9-1e76ec6f883a-kube-api-access-w4bbg" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662223    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63498c86-702e-47d4-8002-0693eb34b05e" volumeName="kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662233    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9de66365-30cc-4adc-ad1b-4a6778246af8" volumeName="kubernetes.io/projected/9de66365-30cc-4adc-ad1b-4a6778246af8-kube-api-access-5cj5d" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662259    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e00a6bb6-263b-40c7-977d-11511832d40a" volumeName="kubernetes.io/projected/e00a6bb6-263b-40c7-977d-11511832d40a-kube-api-access-jd8sr" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662289    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets" seLinuxMountContext=""
Oct 30 03:02:51.662365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662297    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" volumeName="kubernetes.io/secret/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-serving-cert" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662609    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49f31c-2187-47b3-89d9-1e76ec6f883a" volumeName="kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-snapshots" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662647    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c1080c7-2fbb-471b-9ea9-b05fc60f95e9" volumeName="kubernetes.io/configmap/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-config-volume" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662655    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c1080c7-2fbb-471b-9ea9-b05fc60f95e9" volumeName="kubernetes.io/projected/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-kube-api-access-8krn5" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662675    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a35010af-87bc-4a97-81b6-6803ffd77e25" volumeName="kubernetes.io/secret/a35010af-87bc-4a97-81b6-6803ffd77e25-cert" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662715    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662726    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49f31c-2187-47b3-89d9-1e76ec6f883a" volumeName="kubernetes.io/secret/3e49f31c-2187-47b3-89d9-1e76ec6f883a-serving-cert" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662745    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f7168c-a3bc-43e5-a198-b3551b33b770" volumeName="kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662758    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662773    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9da977d0-2885-4889-a18f-9038e7e11272" volumeName="kubernetes.io/configmap/9da977d0-2885-4889-a18f-9038e7e11272-serviceca" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662784    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af46b57a-5155-4ddd-90d8-bf2eb36f8319" volumeName="kubernetes.io/projected/af46b57a-5155-4ddd-90d8-bf2eb36f8319-kube-api-access-8z5dm" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662801    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="060b1735-396e-43d2-940d-e5b9bd3ec06b" volumeName="kubernetes.io/projected/060b1735-396e-43d2-940d-e5b9bd3ec06b-kube-api-access-49rvh" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662814    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662827    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="940bd241-206c-44c1-9c91-b6eb875bce7c" volumeName="kubernetes.io/configmap/940bd241-206c-44c1-9c91-b6eb875bce7c-konnectivity-ca" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662853    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9de66365-30cc-4adc-ad1b-4a6778246af8" volumeName="kubernetes.io/empty-dir/9de66365-30cc-4adc-ad1b-4a6778246af8-tmp-dir" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662867    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7114a74-db33-4c40-a401-030ffa957f21" volumeName="kubernetes.io/projected/b7114a74-db33-4c40-a401-030ffa957f21-kube-api-access-k5pzb" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662884    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec84fda-07e9-469d-bec4-16e9586fed14" volumeName="kubernetes.io/configmap/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-config" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662896    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662910    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3790bf39-be15-4435-8e27-f59fd78ce1af" volumeName="kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662931    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8046eaad-8e51-49d2-beac-8f6be5170f4d" volumeName="kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-tmp" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662944    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c1080c7-2fbb-471b-9ea9-b05fc60f95e9" volumeName="kubernetes.io/empty-dir/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-tmp-dir" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662961    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a35010af-87bc-4a97-81b6-6803ffd77e25" volumeName="kubernetes.io/projected/a35010af-87bc-4a97-81b6-6803ffd77e25-kube-api-access-fz7ct" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662972    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e" volumeName="kubernetes.io/projected/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-api-access-b494s" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662982    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba135d3a-8995-4181-a315-51ae998bba50" volumeName="kubernetes.io/configmap/ba135d3a-8995-4181-a315-51ae998bba50-nginx-conf" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662992    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e00a6bb6-263b-40c7-977d-11511832d40a" volumeName="kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-multus-daemon-config" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663014    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6a644f9-7b06-4072-8052-704244dd5a8d" volumeName="kubernetes.io/projected/f6a644f9-7b06-4072-8052-704244dd5a8d-kube-api-access-qbmg2" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663027    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" volumeName="kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-config" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663039    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63498c86-702e-47d4-8002-0693eb34b05e" volumeName="kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-config" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663051    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63498c86-702e-47d4-8002-0693eb34b05e" volumeName="kubernetes.io/projected/63498c86-702e-47d4-8002-0693eb34b05e-kube-api-access-rv4lv" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663069    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f7168c-a3bc-43e5-a198-b3551b33b770" volumeName="kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663082    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f7168c-a3bc-43e5-a198-b3551b33b770" volumeName="kubernetes.io/projected/66f7168c-a3bc-43e5-a198-b3551b33b770-kube-api-access-fjjfw" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663102    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8046eaad-8e51-49d2-beac-8f6be5170f4d" volumeName="kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-tuned" seLinuxMountContext=""
Oct 30 03:02:51.663169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.662194    4877 factory.go:223] Registration of the systemd container factory successfully
Oct 30 03:02:51.664545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.664309    4877 factory.go:153] Registering CRI-O factory
Oct 30 03:02:51.664545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.664323    4877 factory.go:223] Registration of the crio container factory successfully
Oct 30 03:02:51.664545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.664372    4877 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 03:02:51.664545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.664397    4877 factory.go:103] Registering Raw factory
Oct 30 03:02:51.664545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.664410    4877 manager.go:1199] Started watching for new ooms in manager
Oct 30 03:02:51.664782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.664768    4877 manager.go:319] Starting recovery of all containers
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.663118    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667233    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667257    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" volumeName="kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-trusted-ca" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667279    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f7168c-a3bc-43e5-a198-b3551b33b770" volumeName="kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-binary-copy" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667295    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e" volumeName="kubernetes.io/empty-dir/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-data-volume" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667308    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a767b5d-c7e1-4178-8793-4bd9f4352745" volumeName="kubernetes.io/projected/9a767b5d-c7e1-4178-8793-4bd9f4352745-kube-api-access-x2vqp" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667328    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1d1d308-d556-4dbc-bd3a-2ec2c838413a" volumeName="kubernetes.io/projected/d1d1d308-d556-4dbc-bd3a-2ec2c838413a-kube-api-access-22fcp" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667343    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667364    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29f57388-aede-46d4-9af1-075947d71aa3" volumeName="kubernetes.io/projected/29f57388-aede-46d4-9af1-075947d71aa3-kube-api-access-9k2mp" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667378    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74e50c74-23f8-46b7-98e6-6ac103daba15" volumeName="kubernetes.io/configmap/74e50c74-23f8-46b7-98e6-6ac103daba15-telemetry-config" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667392    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" volumeName="kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667412    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e29e28e-149b-4e87-af1c-a1ab78aa1969" volumeName="kubernetes.io/secret/1e29e28e-149b-4e87-af1c-a1ab78aa1969-serving-cert" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667450    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d8497ef-8d61-4e71-924e-ed835a093f9d" volumeName="kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667469    4877 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e49f31c-2187-47b3-89d9-1e76ec6f883a" volumeName="kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667482    4877 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 03:02:51.668736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.667490    4877 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 03:02:51.679013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.678986    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/1.log"
Oct 30 03:02:51.679340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.679322    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/1.log"
Oct 30 03:02:51.687546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.687510    4877 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 03:02:51.689802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.689782    4877 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 03:02:51.689802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.689806    4877 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 03:02:51.689963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.689823    4877 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 03:02:51.689963 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:51.689855    4877 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 03:02:51.707602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707575    4877 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:51.707759 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707720    4877 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:51.707759 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707726    4877 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 03:02:51.707903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707877    4877 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="0848a71801dc5f5e6fcb668ce792c38b97670cd3bc947f02f7eb3671770e6720" exitCode=0
Oct 30 03:02:51.707903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707899    4877 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="45b24d2c8a442535cc05b1c80bcf6403ee635040ae00c7b81a692243f94a0a48" exitCode=0
Oct 30 03:02:51.708042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707908    4877 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="17f725d0431edd4fa903f2d99568c91b0e8315dbf937a08cf90591115c189166" exitCode=0
Oct 30 03:02:51.708042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707916    4877 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="2b850a08644bd591473705115b3391950acc840759296c0c8b26aa5761d1c69c" exitCode=0
Oct 30 03:02:51.708042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707922    4877 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="1df978a25c2fc1c48b887497ffae6f3fa9136f00a8c931a690d4b83828a7c86f" exitCode=0
Oct 30 03:02:51.708042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.707927    4877 generic.go:356] "Generic (PLEG): container finished" podID="66f7168c-a3bc-43e5-a198-b3551b33b770" containerID="f5a62a8f17b141add81f3cec4d6e990bcebd217c7c7309468508144b6d70a1a1" exitCode=0
Oct 30 03:02:51.720063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.720040    4877 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:51.720223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.720204    4877 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 03:02:51.730692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.730667    4877 generic.go:356] "Generic (PLEG): container finished" podID="4be6db401bef7646097e7da845f451bd" containerID="bfe27021a8036afec71ad276215bfd1c60838152b1b61bcc07792c81ca049539" exitCode=0
Oct 30 03:02:51.735239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.735222    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/1.log"
Oct 30 03:02:51.735344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.735250    4877 generic.go:356] "Generic (PLEG): container finished" podID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a" exitCode=255
Oct 30 03:02:51.751354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.751336    4877 manager.go:324] Recovery completed
Oct 30 03:02:51.772272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772251    4877 cpu_manager.go:226] "Starting" policy="none"
Oct 30 03:02:51.772272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772266    4877 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 03:02:51.772443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772286    4877 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 03:02:51.772500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772466    4877 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 03:02:51.772500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772481    4877 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 03:02:51.772651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772637    4877 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 03:02:51.772712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772652    4877 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 03:02:51.772712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772660    4877 policy_none.go:50] "Start"
Oct 30 03:02:51.772712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772672    4877 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 03:02:51.772712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.772686    4877 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 03:02:51.781047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.781028    4877 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 03:02:51.781150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.781049    4877 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 03:02:51.781150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.781058    4877 policy_none.go:44] "Start"
Oct 30 03:02:51.785226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.785192    4877 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 03:02:51.787644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.787629    4877 manager.go:341] "Starting Device Plugin manager"
Oct 30 03:02:51.788096 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:51.787654    4877 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 03:02:51.788096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788094    4877 server.go:87] "Starting device plugin registration server"
Oct 30 03:02:51.788338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788326    4877 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 03:02:51.788411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788337    4877 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 03:02:51.788486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788442    4877 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 03:02:51.788583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788548    4877 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 03:02:51.788644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788597    4877 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 03:02:51.788644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788604    4877 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 03:02:51.788738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.788660    4877 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:02:51.788580645Z","UUID":"cd9fd3ec-ce34-4b17-bff5-0729e0d9a691","Handler":null,"Name":"","Endpoint":""}
Oct 30 03:02:51.790001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.789980    4877 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 03:02:51.790101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.790010    4877 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 03:02:51.790392 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:51.790366    4877 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 03:02:51.790619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.790572    4877 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"]
Oct 30 03:02:51.791858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.791792    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" event={"ID":"2c7f2093544cf1327cc6ce2b1dd9d362","Type":"ContainerStarted","Data":"e4f20c083e03336efa24a8911bbf1bb98bd03cc7eb93c119f4ae9e50c1e62e52"}
Oct 30 03:02:51.791945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.791873    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" event={"ID":"2c7f2093544cf1327cc6ce2b1dd9d362","Type":"ContainerStarted","Data":"65459d71c0cb2cb3449640da13f0768d5afeb9742632d027a1cde8fcef73de42"}
Oct 30 03:02:51.792022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.792008    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"4be6db401bef7646097e7da845f451bd","Type":"ContainerStarted","Data":"ce51db10fedc2c4d27ed20e681fb052a79d451e1dd7568b2735a7bc97d581ba0"}
Oct 30 03:02:51.792071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.792043    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"4be6db401bef7646097e7da845f451bd","Type":"ContainerDied","Data":"bfe27021a8036afec71ad276215bfd1c60838152b1b61bcc07792c81ca049539"}
Oct 30 03:02:51.792071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.792066    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal" event={"ID":"4be6db401bef7646097e7da845f451bd","Type":"ContainerStarted","Data":"b609ae0b95cd4d2931e19bb72586036f7688d730bb1529bef427051705f4db8b"}
Oct 30 03:02:51.792317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.792292    4877 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.792558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.792534    4877 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.816199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.816176    4877 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:02:51.816297 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:51.816232    4877 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.816816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.816805    4877 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:02:51.816853 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:51.816827    4877 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.889218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.889192    4877 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 03:02:51.890385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.890367    4877 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 03:02:51.890482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.890398    4877 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 03:02:51.890482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.890409    4877 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-226.ec2.internal" event="NodeHasSufficientPID"
Oct 30 03:02:51.890482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.890467    4877 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.908617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.908595    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c7f2093544cf1327cc6ce2b1dd9d362-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2c7f2093544cf1327cc6ce2b1dd9d362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.908672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.908624    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.908672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.908644    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.926226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.926204    4877 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:51.926292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:51.926257    4877 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.009272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.009239    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.009406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.009277    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.009406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.009295    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c7f2093544cf1327cc6ce2b1dd9d362-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2c7f2093544cf1327cc6ce2b1dd9d362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.009406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.009333    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2c7f2093544cf1327cc6ce2b1dd9d362-config\") pod \"kube-apiserver-proxy-ip-10-0-135-226.ec2.internal\" (UID: \"2c7f2093544cf1327cc6ce2b1dd9d362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.009406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.009332    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.009406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.009343    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4be6db401bef7646097e7da845f451bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" (UID: \"4be6db401bef7646097e7da845f451bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.644199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.644165    4877 apiserver.go:52] "Watching apiserver"
Oct 30 03:02:52.696562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.696533    4877 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 03:02:52.697346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.697319    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lrntf","kube-system/kas-connection-checker-685895c7c9-zwfb4","kube-system/konnectivity-agent-sfv5t","openshift-insights/insights-operator-7c4986d449-klw22","openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v","openshift-console-operator/console-operator-7cd56b8dbc-qjtcv","openshift-dns/node-resolver-2hl5n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal","openshift-ingress/router-default-59ddf86b9f-2wfp9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw","openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk","openshift-dns/dns-default-cfphv","kube-system/kas-connection-checker-685895c7c9-gt4dd","openshift-network-operator/iptables-alerter-2qcvz","kube-system/kas-connection-checker-685895c7c9-ptzxj","openshift-multus/multus-czcg4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr","openshift-image-registry/image-registry-7cd96bbd68-84shk","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s","openshift-ingress-canary/ingress-canary-ptqz4","kube-system/global-pull-secret-syncer-rtd54","openshift-insights/insights-runtime-extractor-9xq4n","kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-z54vn","openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m","openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9","openshift-multus/multus-additional-cni-plugins-8wqsd","openshift-cluster-node-tuning-operator/tuned-ljjrp","openshift-image-registry/node-ca-z5sbs","openshift-multus/network-metrics-daemon-z74kk","openshift-network-diagnostics/network-check-target-6ssh4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"]
Oct 30 03:02:52.701787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.701762    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.702109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.702091    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.702792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.702776    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.703655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.703639    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.704366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.704338    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:52.706152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.706132    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:52.706484 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.706469    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.710615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.710599    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.711068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.711051    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.712572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.712556    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.712572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.712568    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.716859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716833    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8g9pb\""
Oct 30 03:02:52.716859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716835    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 03:02:52.717019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716869    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.717019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716896    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.717019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716915    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.717019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716937    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 03:02:52.717019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.716956    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.717176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717045    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tpt7c\""
Oct 30 03:02:52.717176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717145    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 03:02:52.717248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717200    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.717351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717333    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 03:02:52.717404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717391    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.717798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717781    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cq8qc\""
Oct 30 03:02:52.717850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717803    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 03:02:52.717850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.717804    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 03:02:52.724222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.724208    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.725976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.725960    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 03:02:52.726206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726188    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726196    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726192    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726221    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726231    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726254    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726285    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726296    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726300    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726303    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726295    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.726322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726316    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.726673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726344    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.726673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.726347    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.727617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727601    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 03:02:52.727684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727635    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.727854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727840    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.727905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727860    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.727905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727868    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.727905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727860    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.727905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.727896    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.728116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.728105    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.730138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730012    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t77t7\""
Oct 30 03:02:52.730138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730027    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.730138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730068    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 03:02:52.730138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730079    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.730138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730077    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.730138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730019    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 03:02:52.730476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730145    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 03:02:52.730476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730190    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 03:02:52.730476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730256    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 03:02:52.730476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730325    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-r8pw4\""
Oct 30 03:02:52.730476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730361    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-njwkw\""
Oct 30 03:02:52.730476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730417    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730496    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730504    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730504    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730530    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730532    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730556    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wcx8m\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730559    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whjh9\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730616    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730671    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.730755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730732    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730766    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730781    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730823    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mpbq7\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730847    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730857    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7q2t\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730976    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6fpwt\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.730993    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731001    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731003    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnh6p\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731019    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731013    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731037    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731047    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731058    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f9hw2\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731083    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgj97\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731090    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nc7qz\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731101    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731110    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731119    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731128    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731150    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qj6bv\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731164    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731169    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731183    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 03:02:52.731189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731201    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731210    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731231    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731204    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-69wt9\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731214    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bl9p4\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731251    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731280    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r5bpm\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731352    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 03:02:52.731957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.731436    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-shnp4\""
Oct 30 03:02:52.734235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.734218    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hhqhj\""
Oct 30 03:02:52.734300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.734277    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dt8fk\""
Oct 30 03:02:52.736641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.736626    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 03:02:52.736706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.736693    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 03:02:52.736751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.736714    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j94bv\""
Oct 30 03:02:52.738090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.738075    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 03:02:52.738335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.738323    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-scdl7\""
Oct 30 03:02:52.739762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.739747    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vnb2g\""
Oct 30 03:02:52.739969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.739956    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc5sd\""
Oct 30 03:02:52.740018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.739958    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 03:02:52.740386 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.740372    4877 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.760081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.760051    4877 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 03:02:52.761464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.761448    4877 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 03:02:52.761534 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.761497    4877 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-226.ec2.internal"
Oct 30 03:02:52.812797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.812772    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-tmp\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.812891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.812806    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.812891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.812832    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-conf-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.813005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.812903    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-registration-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.813005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.812943    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-tmp\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.813005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.812997    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkx9\" (UniqueName: \"kubernetes.io/projected/fdce563b-9436-449d-8cd6-159e7da40143-kube-api-access-zjkx9\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.813110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813041    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-system-cni-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.813187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813168    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.813221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813202    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.813273 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813229    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.813273 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813255    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.813366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813273    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-modprobe-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.813366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813287    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-run\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.813366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813306    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4bs\" (UniqueName: \"kubernetes.io/projected/8046eaad-8e51-49d2-beac-8f6be5170f4d-kube-api-access-sh4bs\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.813366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813331    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-crio-socket\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.813572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813449    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-system-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.813572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813475    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.813572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813481    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.813572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813492    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.813572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813534    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqq7j\" (UniqueName: \"kubernetes.io/projected/160c1f29-a378-43d0-97d8-be1cc643e2c2-kube-api-access-bqq7j\") pod \"kas-connection-checker-685895c7c9-zwfb4\" (UID: \"160c1f29-a378-43d0-97d8-be1cc643e2c2\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813576    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b494s\" (UniqueName: \"kubernetes.io/projected/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-api-access-b494s\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813592    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813602    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/940bd241-206c-44c1-9c91-b6eb875bce7c-konnectivity-ca\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813634    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-multus\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813650    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krn5\" (UniqueName: \"kubernetes.io/projected/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-kube-api-access-8krn5\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813667    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-cnibin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813683    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-kubelet\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813720    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-socket-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.813752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813749    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-tmp\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813773    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8sr\" (UniqueName: \"kubernetes.io/projected/e00a6bb6-263b-40c7-977d-11511832d40a-kube-api-access-jd8sr\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813816    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-netns\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813847    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813855    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-tmp\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813870    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35010af-87bc-4a97-81b6-6803ffd77e25-cert\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813903    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplzq\" (UniqueName: \"kubernetes.io/projected/74e50c74-23f8-46b7-98e6-6ac103daba15-kube-api-access-nplzq\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.813927    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-slash\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814007    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/940bd241-206c-44c1-9c91-b6eb875bce7c-konnectivity-ca\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:52.814123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814111    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4lv\" (UniqueName: \"kubernetes.io/projected/63498c86-702e-47d4-8002-0693eb34b05e-kube-api-access-rv4lv\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.814478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814134    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a35010af-87bc-4a97-81b6-6803ffd77e25-cert\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:52.814478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814137    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bbg\" (UniqueName: \"kubernetes.io/projected/3e49f31c-2187-47b3-89d9-1e76ec6f883a-kube-api-access-w4bbg\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.814478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814164    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.814478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814237    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-systemd\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.814478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814337    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fcp\" (UniqueName: \"kubernetes.io/projected/d1d1d308-d556-4dbc-bd3a-2ec2c838413a-kube-api-access-22fcp\") pod \"kas-connection-checker-685895c7c9-ptzxj\" (UID: \"d1d1d308-d556-4dbc-bd3a-2ec2c838413a\") " pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:52.814478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814461    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rvh\" (UniqueName: \"kubernetes.io/projected/060b1735-396e-43d2-940d-e5b9bd3ec06b-kube-api-access-49rvh\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:52.814755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814552    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.814755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814612    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-trusted-ca\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.814755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814633    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.814755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814653    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-default-certificate\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.814755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814674    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:52.814755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814723    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjfw\" (UniqueName: \"kubernetes.io/projected/66f7168c-a3bc-43e5-a198-b3551b33b770-kube-api-access-fjjfw\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814757    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-node-log\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814782    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-bin\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814841    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-config\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814875    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysconfig\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814900    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-default-certificate\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814901    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-var-lib-kubelet\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814964    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-trusted-ca\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.815020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.814974    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pzb\" (UniqueName: \"kubernetes.io/projected/b7114a74-db33-4c40-a401-030ffa957f21-kube-api-access-k5pzb\") pod \"volume-data-source-validator-58c9f5f46d-2mdnr\" (UID: \"b7114a74-db33-4c40-a401-030ffa957f21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815090    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-systemd-units\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815117    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63498c86-702e-47d4-8002-0693eb34b05e-ovn-node-metrics-cert\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815137    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-service-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815161    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-etc-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815165    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-config\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815274    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63498c86-702e-47d4-8002-0693eb34b05e-ovn-node-metrics-cert\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.815337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815299    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-config-volume\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.815562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815377    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-service-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.815562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815483    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.815562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815539    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-host\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.815680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815573    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-config-volume\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.815680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815580    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.815680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815581    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.815680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815616    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de66365-30cc-4adc-ad1b-4a6778246af8-tmp-dir\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.815680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815659    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.815898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815703    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a767b5d-c7e1-4178-8793-4bd9f4352745-iptables-alerter-script\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.815898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815741    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de66365-30cc-4adc-ad1b-4a6778246af8-tmp-dir\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.815898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815786    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vqp\" (UniqueName: \"kubernetes.io/projected/9a767b5d-c7e1-4178-8793-4bd9f4352745-kube-api-access-x2vqp\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.815898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815812    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-sys-fs\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.815898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815837    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-data-volume\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.815898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815860    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-kubelet-config\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.816092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815922    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-multus-daemon-config\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.816092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815968    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-env-overrides\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.816092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.815995    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-sys\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.816092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816016    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-stats-auth\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.816092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816080    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.816222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816095    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a767b5d-c7e1-4178-8793-4bd9f4352745-iptables-alerter-script\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.816222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816100    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cj5d\" (UniqueName: \"kubernetes.io/projected/9de66365-30cc-4adc-ad1b-4a6778246af8-kube-api-access-5cj5d\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.816222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816141    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-data-volume\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.816222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816166    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-config\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.816326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816269    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6a644f9-7b06-4072-8052-704244dd5a8d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:52.816326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816285    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-script-lib\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.816326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816301    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.816326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816319    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.816452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816348    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-multus-daemon-config\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.816452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816367    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.816452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816385    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-serving-cert\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.816452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816389    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-config\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.816452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816437    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-kubelet\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816456    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a644f9-7b06-4072-8052-704244dd5a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-netd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816516    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrj2\" (UniqueName: \"kubernetes.io/projected/bec84fda-07e9-469d-bec4-16e9586fed14-kube-api-access-kjrj2\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816514    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-env-overrides\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816533    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-bin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816348    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-stats-auth\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816549    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74e50c74-23f8-46b7-98e6-6ac103daba15-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.816633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816612    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816639    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-serving-cert\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816661    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km82f\" (UniqueName: \"kubernetes.io/projected/1569b662-064a-4998-b332-03c6ed48e312-kube-api-access-km82f\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816685    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7ct\" (UniqueName: \"kubernetes.io/projected/a35010af-87bc-4a97-81b6-6803ffd77e25-kube-api-access-fz7ct\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816708    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8549h\" (UniqueName: \"kubernetes.io/projected/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-kube-api-access-8549h\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816707    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816727    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-netns\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.816839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816733    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74e50c74-23f8-46b7-98e6-6ac103daba15-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816846    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-multus-certs\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816865    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63498c86-702e-47d4-8002-0693eb34b05e-ovnkube-script-lib\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816892    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-ovn\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816922    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-log-socket\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816951    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9da977d0-2885-4889-a18f-9038e7e11272-serviceca\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816956    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6a644f9-7b06-4072-8052-704244dd5a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816951    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.817030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.816982    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.817283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817104    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6a644f9-7b06-4072-8052-704244dd5a8d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:52.817283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817172    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/af46b57a-5155-4ddd-90d8-bf2eb36f8319-kube-api-access-8z5dm\") pod \"kas-connection-checker-685895c7c9-gt4dd\" (UID: \"af46b57a-5155-4ddd-90d8-bf2eb36f8319\") " pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:52.817283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817195    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:52.817283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817230    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-lib-modules\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.817283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817249    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-cnibin\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.817283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817267    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817296    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9da977d0-2885-4889-a18f-9038e7e11272-serviceca\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817307    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-etc-kubernetes\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817326    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtjx\" (UniqueName: \"kubernetes.io/projected/9da977d0-2885-4889-a18f-9038e7e11272-kube-api-access-8xtjx\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817342    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-conf\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817360    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817399    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a767b5d-c7e1-4178-8793-4bd9f4352745-host-slash\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817416    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817450    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817472    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmg2\" (UniqueName: \"kubernetes.io/projected/f6a644f9-7b06-4072-8052-704244dd5a8d-kube-api-access-qbmg2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817489    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817501    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817535    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817553    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e29e28e-149b-4e87-af1c-a1ab78aa1969-serving-cert\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817572    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e29e28e-149b-4e87-af1c-a1ab78aa1969-config\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817589    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-var-lib-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:52.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817625    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817666    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-socket-dir-parent\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817712    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-tmp-dir\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817747    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-device-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817784    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba135d3a-8995-4181-a315-51ae998bba50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817826    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817850    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817883    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.817927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817909    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-cni-binary-copy\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.818138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817936    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.818138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817959    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e29e28e-149b-4e87-af1c-a1ab78aa1969-serving-cert\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:52.818138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.817971    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-snapshots\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.818138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818006    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.818138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818039    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.818138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818088    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e29e28e-149b-4e87-af1c-a1ab78aa1969-config\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:52.818338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818204    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-tmp-dir\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.818338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818220    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.818338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818252    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hps\" (UniqueName: \"kubernetes.io/projected/1e29e28e-149b-4e87-af1c-a1ab78aa1969-kube-api-access-p2hps\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:52.818338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818285    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-systemd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.818338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818310    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.818338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818320    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.818511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818348    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2mp\" (UniqueName: \"kubernetes.io/projected/29f57388-aede-46d4-9af1-075947d71aa3-kube-api-access-9k2mp\") pod \"network-check-source-6f78d59f99-zn99v\" (UID: \"29f57388-aede-46d4-9af1-075947d71aa3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:52.818511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818368    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e49f31c-2187-47b3-89d9-1e76ec6f883a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.818511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818450    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f7168c-a3bc-43e5-a198-b3551b33b770-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.818511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818456    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-dbus\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.818511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818490    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818519    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e49f31c-2187-47b3-89d9-1e76ec6f883a-snapshots\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818521    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9de66365-30cc-4adc-ad1b-4a6778246af8-hosts-file\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818550    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00a6bb6-263b-40c7-977d-11511832d40a-cni-binary-copy\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818577    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-k8s-cni-cncf-io\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818595    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-hostroot\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818612    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818632    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818646    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da977d0-2885-4889-a18f-9038e7e11272-host\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.818675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818661    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-kubernetes\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.818909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818779    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bce76cad-b673-431c-b6a4-40298db25172-original-pull-secret\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.818909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818857    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.818909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818879    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.818909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818896    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e49f31c-2187-47b3-89d9-1e76ec6f883a-serving-cert\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.819015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818914    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-tuned\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.819015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818931    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.819015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818947    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.819015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.818966    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/940bd241-206c-44c1-9c91-b6eb875bce7c-agent-certs\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:52.819190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.819173    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/940bd241-206c-44c1-9c91-b6eb875bce7c-agent-certs\") pod \"konnectivity-agent-sfv5t\" (UID: \"940bd241-206c-44c1-9c91-b6eb875bce7c\") " pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:02:52.819312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.819301    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.819413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.819402    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e49f31c-2187-47b3-89d9-1e76ec6f883a-serving-cert\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.819467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.819415    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba135d3a-8995-4181-a315-51ae998bba50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:52.819502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.819486    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-tuned\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.833187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.832933    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b494s\" (UniqueName: \"kubernetes.io/projected/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-api-access-b494s\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.833187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833076    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkx9\" (UniqueName: \"kubernetes.io/projected/fdce563b-9436-449d-8cd6-159e7da40143-kube-api-access-zjkx9\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.833187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833147    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krn5\" (UniqueName: \"kubernetes.io/projected/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-kube-api-access-8krn5\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.833187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833176    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqq7j\" (UniqueName: \"kubernetes.io/projected/160c1f29-a378-43d0-97d8-be1cc643e2c2-kube-api-access-bqq7j\") pod \"kas-connection-checker-685895c7c9-zwfb4\" (UID: \"160c1f29-a378-43d0-97d8-be1cc643e2c2\") " pod="kube-system/kas-connection-checker-685895c7c9-zwfb4"
Oct 30 03:02:52.833404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833218    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8sr\" (UniqueName: \"kubernetes.io/projected/e00a6bb6-263b-40c7-977d-11511832d40a-kube-api-access-jd8sr\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.833404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833338    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fcp\" (UniqueName: \"kubernetes.io/projected/d1d1d308-d556-4dbc-bd3a-2ec2c838413a-kube-api-access-22fcp\") pod \"kas-connection-checker-685895c7c9-ptzxj\" (UID: \"d1d1d308-d556-4dbc-bd3a-2ec2c838413a\") " pod="kube-system/kas-connection-checker-685895c7c9-ptzxj"
Oct 30 03:02:52.833404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833342    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4bs\" (UniqueName: \"kubernetes.io/projected/8046eaad-8e51-49d2-beac-8f6be5170f4d-kube-api-access-sh4bs\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.833779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833740    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjfw\" (UniqueName: \"kubernetes.io/projected/66f7168c-a3bc-43e5-a198-b3551b33b770-kube-api-access-fjjfw\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.833870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833831    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rvh\" (UniqueName: \"kubernetes.io/projected/060b1735-396e-43d2-940d-e5b9bd3ec06b-kube-api-access-49rvh\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:52.833870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833842    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4lv\" (UniqueName: \"kubernetes.io/projected/63498c86-702e-47d4-8002-0693eb34b05e-kube-api-access-rv4lv\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.833979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833936    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.834032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.833996    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pzb\" (UniqueName: \"kubernetes.io/projected/b7114a74-db33-4c40-a401-030ffa957f21-kube-api-access-k5pzb\") pod \"volume-data-source-validator-58c9f5f46d-2mdnr\" (UID: \"b7114a74-db33-4c40-a401-030ffa957f21\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2mdnr"
Oct 30 03:02:52.834087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.834068    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bbg\" (UniqueName: \"kubernetes.io/projected/3e49f31c-2187-47b3-89d9-1e76ec6f883a-kube-api-access-w4bbg\") pod \"insights-operator-7c4986d449-klw22\" (UID: \"3e49f31c-2187-47b3-89d9-1e76ec6f883a\") " pod="openshift-insights/insights-operator-7c4986d449-klw22"
Oct 30 03:02:52.834141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.834099    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplzq\" (UniqueName: \"kubernetes.io/projected/74e50c74-23f8-46b7-98e6-6ac103daba15-kube-api-access-nplzq\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.834707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.834688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cj5d\" (UniqueName: \"kubernetes.io/projected/9de66365-30cc-4adc-ad1b-4a6778246af8-kube-api-access-5cj5d\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.835091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.835072    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vqp\" (UniqueName: \"kubernetes.io/projected/9a767b5d-c7e1-4178-8793-4bd9f4352745-kube-api-access-x2vqp\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.835315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.835298    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrj2\" (UniqueName: \"kubernetes.io/projected/bec84fda-07e9-469d-bec4-16e9586fed14-kube-api-access-kjrj2\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.835702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.835684    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.837058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837036    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8549h\" (UniqueName: \"kubernetes.io/projected/3a65983f-48e3-4e8f-a576-2110dfbb2ec2-kube-api-access-8549h\") pod \"console-operator-7cd56b8dbc-qjtcv\" (UID: \"3a65983f-48e3-4e8f-a576-2110dfbb2ec2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:52.837326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837177    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmg2\" (UniqueName: \"kubernetes.io/projected/f6a644f9-7b06-4072-8052-704244dd5a8d-kube-api-access-qbmg2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4\" (UID: \"f6a644f9-7b06-4072-8052-704244dd5a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4"
Oct 30 03:02:52.837521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837494    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.837619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837530    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssbw\" (UniqueName: \"kubernetes.io/projected/3790bf39-be15-4435-8e27-f59fd78ce1af-kube-api-access-xssbw\") pod \"network-check-target-6ssh4\" (UID: \"3790bf39-be15-4435-8e27-f59fd78ce1af\") " pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:52.837677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837634    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km82f\" (UniqueName: \"kubernetes.io/projected/1569b662-064a-4998-b332-03c6ed48e312-kube-api-access-km82f\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.837869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837850    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2mp\" (UniqueName: \"kubernetes.io/projected/29f57388-aede-46d4-9af1-075947d71aa3-kube-api-access-9k2mp\") pod \"network-check-source-6f78d59f99-zn99v\" (UID: \"29f57388-aede-46d4-9af1-075947d71aa3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-zn99v"
Oct 30 03:02:52.837981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.837958    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtjx\" (UniqueName: \"kubernetes.io/projected/9da977d0-2885-4889-a18f-9038e7e11272-kube-api-access-8xtjx\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.838150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.838134    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hps\" (UniqueName: \"kubernetes.io/projected/1e29e28e-149b-4e87-af1c-a1ab78aa1969-kube-api-access-p2hps\") pod \"service-ca-operator-544d749c8c-plhvk\" (UID: \"1e29e28e-149b-4e87-af1c-a1ab78aa1969\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk"
Oct 30 03:02:52.838150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.838141    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5dm\" (UniqueName: \"kubernetes.io/projected/af46b57a-5155-4ddd-90d8-bf2eb36f8319-kube-api-access-8z5dm\") pod \"kas-connection-checker-685895c7c9-gt4dd\" (UID: \"af46b57a-5155-4ddd-90d8-bf2eb36f8319\") " pod="kube-system/kas-connection-checker-685895c7c9-gt4dd"
Oct 30 03:02:52.838287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.838268    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7ct\" (UniqueName: \"kubernetes.io/projected/a35010af-87bc-4a97-81b6-6803ffd77e25-kube-api-access-fz7ct\") pod \"ingress-canary-ptqz4\" (UID: \"a35010af-87bc-4a97-81b6-6803ffd77e25\") " pod="openshift-ingress-canary/ingress-canary-ptqz4"
Oct 30 03:02:52.838318 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.838274    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.919258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919196    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919233    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:52.919258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919257    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-node-log\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919277    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-bin\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919301    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919332    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-bin\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919337    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-node-log\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919363    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysconfig\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919386    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-var-lib-kubelet\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.919348    4877 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919406    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysconfig\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919437    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-systemd-units\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919469    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-var-lib-kubelet\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919480    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-systemd-units\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919472    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-etc-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.919491    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.419470009 +0000 UTC m=+1.848359421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : secret "metrics-daemon-secret" not found
Oct 30 03:02:52.919515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919510    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-etc-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919549    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-host\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919578    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-sys-fs\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919601    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-kubelet-config\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919634    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-sys\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919671    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-kubelet\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919677    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-kubelet-config\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-sys-fs\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919642    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-host\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919709    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-netd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919721    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-sys\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919737    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-cni-netd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919740    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-bin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919737    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-kubelet\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919766    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-netns\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919789    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-multus-certs\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919807    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-bin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919828    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-netns\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919813    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-ovn\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919852    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-ovn\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919867    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-log-socket\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919870    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-multus-certs\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919885    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919902    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919919    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-lib-modules\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919935    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-cnibin\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919953    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-log-socket\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919957    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-etc-kubernetes\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.919984    4877 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919987    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-conf\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920014    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920021    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-cnibin\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920046    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.4200303 +0000 UTC m=+1.848919703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.919986    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-etc-kubernetes\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920047    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-lib-modules\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920068    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a767b5d-c7e1-4178-8793-4bd9f4352745-host-slash\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920084    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a767b5d-c7e1-4178-8793-4bd9f4352745-host-slash\") pod \"iptables-alerter-2qcvz\" (UID: \"9a767b5d-c7e1-4178-8793-4bd9f4352745\") " pod="openshift-network-operator/iptables-alerter-2qcvz"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920101    4877 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920103    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.420082419 +0000 UTC m=+1.848971819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920134    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920146    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.420137558 +0000 UTC m=+1.849026974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920144    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-conf\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.920134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920164    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920168    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-var-lib-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920186    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920193    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-var-lib-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920205    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-socket-dir-parent\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920221    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-device-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920236    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920246    4877 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920279    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.420268244 +0000 UTC m=+1.849157662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920301    4877 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920302    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-device-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920303    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-socket-dir-parent\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920326    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.420319632 +0000 UTC m=+1.849209028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920330    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920385    4877 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920393    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-systemd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920413    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.420403295 +0000 UTC m=+1.849292696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920450    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-dbus\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920453    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-systemd\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920467    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9de66365-30cc-4adc-ad1b-4a6778246af8-hosts-file\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920498    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-k8s-cni-cncf-io\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920514    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-hostroot\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920511    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9de66365-30cc-4adc-ad1b-4a6778246af8-hosts-file\") pod \"node-resolver-2hl5n\" (UID: \"9de66365-30cc-4adc-ad1b-4a6778246af8\") " pod="openshift-dns/node-resolver-2hl5n"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920531    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920530    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bce76cad-b673-431c-b6a4-40298db25172-dbus\") pod \"global-pull-secret-syncer-rtd54\" (UID: \"bce76cad-b673-431c-b6a4-40298db25172\") " pod="kube-system/global-pull-secret-syncer-rtd54"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920567    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-hostroot\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920593    4877 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.920622    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.420613155 +0000 UTC m=+1.849502555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920591    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-run-k8s-cni-cncf-io\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920647    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.920561    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921018    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da977d0-2885-4889-a18f-9038e7e11272-host\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921040    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-kubernetes\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921063    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921092    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921113    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921117    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da977d0-2885-4889-a18f-9038e7e11272-host\") pod \"node-ca-z5sbs\" (UID: \"9da977d0-2885-4889-a18f-9038e7e11272\") " pod="openshift-image-registry/node-ca-z5sbs"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921129    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921147    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-conf-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921165    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-registration-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921181    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-system-cni-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921206    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921216    4877 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921221    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-modprobe-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921238    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921257    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921263    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.421248557 +0000 UTC m=+1.850137959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921259    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921261    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-registration-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921279    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f7168c-a3bc-43e5-a198-b3551b33b770-system-cni-dir\") pod \"multus-additional-cni-plugins-8wqsd\" (UID: \"66f7168c-a3bc-43e5-a198-b3551b33b770\") " pod="openshift-multus/multus-additional-cni-plugins-8wqsd"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921243    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-run\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921167    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-kubernetes\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921298    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-run\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921301    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921308    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.421293296 +0000 UTC m=+1.850182712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921326    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-conf-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921343    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-sysctl-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921354    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-crio-socket\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921364    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-modprobe-d\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921384    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-crio-socket\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921384    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-system-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921414    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921452    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-system-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921459    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-multus\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921474    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-cnibin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921488    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-multus-cni-dir\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921494    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-kubelet\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921519    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-kubelet\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921546    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-host-var-lib-cni-multus\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921548    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-socket-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921580    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fdce563b-9436-449d-8cd6-159e7da40143-socket-dir\") pod \"aws-ebs-csi-driver-node-9nbvw\" (UID: \"fdce563b-9436-449d-8cd6-159e7da40143\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nbvw"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921584    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-netns\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921608    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-run-netns\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921553    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00a6bb6-263b-40c7-977d-11511832d40a-cnibin\") pod \"multus-czcg4\" (UID: \"e00a6bb6-263b-40c7-977d-11511832d40a\") " pod="openshift-multus/multus-czcg4"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921616    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921644    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-slash\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921671    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921702    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921702    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-systemd\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921710    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921722    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-run-openvswitch\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921704    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63498c86-702e-47d4-8002-0693eb34b05e-host-slash\") pod \"ovnkube-node-z54vn\" (UID: \"63498c86-702e-47d4-8002-0693eb34b05e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:52.921736    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.421727039 +0000 UTC m=+1.850616470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:52.921895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:52.921793    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8046eaad-8e51-49d2-beac-8f6be5170f4d-etc-systemd\") pod \"tuned-ljjrp\" (UID: \"8046eaad-8e51-49d2-beac-8f6be5170f4d\") " pod="openshift-cluster-node-tuning-operator/tuned-ljjrp"
Oct 30 03:02:53.001750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.001730    4877 scope.go:122] "RemoveContainer" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a"
Oct 30 03:02:53.426222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426186    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:53.426378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426229    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:53.426378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426265    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:53.426378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426282    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:53.426378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426303    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:53.426378 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426321    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:53.426378 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426340    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426387    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426373214 +0000 UTC m=+2.855262633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426387    4877 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426415    4877 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426442    4877 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426323    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426442    4877 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426449    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426418837 +0000 UTC m=+2.855308246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426531    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426515808 +0000 UTC m=+2.855405206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426547    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426540556 +0000 UTC m=+2.855429957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426559    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426553341 +0000 UTC m=+2.855442741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : secret "metrics-daemon-secret" not found
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426573    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426568187 +0000 UTC m=+2.855457587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426603    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426624    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:53.426651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426647    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426666    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.426693    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426731    4877 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426737    4877 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426740    4877 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426770    4877 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426802    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426824    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426806    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426759679 +0000 UTC m=+2.855649090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426868    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426858378 +0000 UTC m=+2.855747774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426878    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426872109 +0000 UTC m=+2.855761505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426891    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426882977 +0000 UTC m=+2.855772373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:53.427151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.426906    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.426897389 +0000 UTC m=+2.855786786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:53.706047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.705981    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2hl5n_9de66365-30cc-4adc-ad1b-4a6778246af8/dns-node-resolver/0.log"
Oct 30 03:02:53.743985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.743957    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:02:53.744370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.744354    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/1.log"
Oct 30 03:02:53.744467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.744388    4877 generic.go:356] "Generic (PLEG): container finished" podID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" containerID="3ae9fe9f7bc13df9ae35a77cd77bab5625c123e57db7df5eeb0ae662675c8a7a" exitCode=255
Oct 30 03:02:53.744513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.744462    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" event={"ID":"3a65983f-48e3-4e8f-a576-2110dfbb2ec2","Type":"ContainerDied","Data":"3ae9fe9f7bc13df9ae35a77cd77bab5625c123e57db7df5eeb0ae662675c8a7a"}
Oct 30 03:02:53.744566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.744525    4877 scope.go:122] "RemoveContainer" containerID="086c9d5bfb34a1bb05f82f18bc48fa26004c9ec902c4647d98c9e55d5912d01a"
Oct 30 03:02:53.744955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:53.744933    4877 scope.go:122] "RemoveContainer" containerID="3ae9fe9f7bc13df9ae35a77cd77bab5625c123e57db7df5eeb0ae662675c8a7a"
Oct 30 03:02:53.745146 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:53.745117    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qjtcv_openshift-console-operator(3a65983f-48e3-4e8f-a576-2110dfbb2ec2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2"
Oct 30 03:02:54.437327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437292    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437332    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437355    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437375    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437394    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437452    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437465    4877 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437479    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437493    4877 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437508    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437523    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437536    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.43751788 +0000 UTC m=+4.866407291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:54.437538 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437542    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437569    4877 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437583    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437592    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437580563 +0000 UTC m=+4.866469958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437599    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437624    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437608673 +0000 UTC m=+4.866498073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437524    4877 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437650    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437638017 +0000 UTC m=+4.866527415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : secret "metrics-daemon-secret" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437466    4877 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437668    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437659663 +0000 UTC m=+4.866549062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437466    4877 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437687    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437678557 +0000 UTC m=+4.866567953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437709    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.4376973 +0000 UTC m=+4.866586707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : secret "router-metrics-certs-default" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437726    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437717905 +0000 UTC m=+4.866607300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437786    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437841    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.437878    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437934    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle podName:1569b662-064a-4998-b332-03c6ed48e312 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437922162 +0000 UTC m=+4.866811578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle") pod "router-default-59ddf86b9f-2wfp9" (UID: "1569b662-064a-4998-b332-03c6ed48e312") : configmap references non-existent config key: service-ca.crt
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437970    4877 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.437978    4877 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.438003    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.437992039 +0000 UTC m=+4.866881455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:54.438017 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:54.438017    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:56.438011053 +0000 UTC m=+4.866900449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:54.749404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:54.749339    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:02:55.019414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:55.019344    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:55.114648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:55.114621    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5sbs_9da977d0-2885-4889-a18f-9038e7e11272/node-ca/0.log"
Oct 30 03:02:55.479540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:55.479507    4877 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:55.479843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:55.479829    4877 scope.go:122] "RemoveContainer" containerID="3ae9fe9f7bc13df9ae35a77cd77bab5625c123e57db7df5eeb0ae662675c8a7a"
Oct 30 03:02:55.480009 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:55.479991    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qjtcv_openshift-console-operator(3a65983f-48e3-4e8f-a576-2110dfbb2ec2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2"
Oct 30 03:02:55.705594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:55.705565    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ptqz4_a35010af-87bc-4a97-81b6-6803ffd77e25/serve-healthcheck-canary/0.log"
Oct 30 03:02:56.458147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458109    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:02:56.458147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458148    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458171    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458199    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458219    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458243    4877 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458259    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458285    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458245    4877 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458307    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls podName:74e50c74-23f8-46b7-98e6-6ac103daba15 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458288856 +0000 UTC m=+8.887178253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ph5d9" (UID: "74e50c74-23f8-46b7-98e6-6ac103daba15") : secret "cluster-monitoring-operator-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458319    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458339    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m: secret "image-registry-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458343    4877 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458348    4877 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458352    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458365    4877 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cd96bbd68-84shk: secret "image-registry-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458374    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458349432 +0000 UTC m=+8.887238844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458352    4877 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458405    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls podName:fcca0b47-f1cc-4b6f-b20d-ebfb450be528 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458392134 +0000 UTC m=+8.887281532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls") pod "image-registry-7bf5b6fbbf-6hq7m" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528") : secret "image-registry-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458407    4877 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458450    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458439196 +0000 UTC m=+8.887328597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : secret "metrics-daemon-secret" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458472    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458462389 +0000 UTC m=+8.887351786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458495    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls podName:2d8497ef-8d61-4e71-924e-ed835a093f9d nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458485562 +0000 UTC m=+8.887374958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls") pod "image-registry-7cd96bbd68-84shk" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d") : secret "image-registry-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458516    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458508349 +0000 UTC m=+8.887397746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458564    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458597    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:02:56.458668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458627    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:02:56.459452 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458707    4877 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:02:56.459452 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458739    4877 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:02:56.459452 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458756    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458742481 +0000 UTC m=+8.887631880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:02:56.459452 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:56.458780    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.458768502 +0000 UTC m=+8.887657904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:02:56.459452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458870    4877 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 03:02:56.459452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.458957    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1569b662-064a-4998-b332-03c6ed48e312-service-ca-bundle\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:56.460926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.460907    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1569b662-064a-4998-b332-03c6ed48e312-metrics-certs\") pod \"router-default-59ddf86b9f-2wfp9\" (UID: \"1569b662-064a-4998-b332-03c6ed48e312\") " pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:56.509888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.509859    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4_f6a644f9-7b06-4072-8052-704244dd5a8d/kube-storage-version-migrator-operator/0.log"
Oct 30 03:02:56.602624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.602601    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:56.909868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:56.909837    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59ddf86b9f-2wfp9"]
Oct 30 03:02:56.910757 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:02:56.910732    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1569b662_064a_4998_b332_03c6ed48e312.slice/crio-0edfb142ba31a290394eb85507c9ae3ab53368e1af4423c4030b981f8c165155 WatchSource:0}: Error finding container 0edfb142ba31a290394eb85507c9ae3ab53368e1af4423c4030b981f8c165155: Status 404 returned error can't find the container with id 0edfb142ba31a290394eb85507c9ae3ab53368e1af4423c4030b981f8c165155
Oct 30 03:02:57.760262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:57.760224    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9" event={"ID":"1569b662-064a-4998-b332-03c6ed48e312","Type":"ContainerStarted","Data":"6157d4b9a5722e506e5afffc1e488392bc80fc3e0ea471cce2b4524f8b1f8db9"}
Oct 30 03:02:57.760757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:57.760276    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9" event={"ID":"1569b662-064a-4998-b332-03c6ed48e312","Type":"ContainerStarted","Data":"0edfb142ba31a290394eb85507c9ae3ab53368e1af4423c4030b981f8c165155"}
Oct 30 03:02:58.366344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:58.366309    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:02:58.366749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:58.366731    4877 scope.go:122] "RemoveContainer" containerID="3ae9fe9f7bc13df9ae35a77cd77bab5625c123e57db7df5eeb0ae662675c8a7a"
Oct 30 03:02:58.366940 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:02:58.366918    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qjtcv_openshift-console-operator(3a65983f-48e3-4e8f-a576-2110dfbb2ec2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2"
Oct 30 03:02:58.602906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:58.602880    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:58.605375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:58.605354    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:02:59.064952 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:59.064920    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:02:59.065332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:59.065118    4877 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:02:59.654839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:59.654813    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:02:59.765106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:02:59.765078    4877 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:03:00.494732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.494696    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:03:00.494732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.494735    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.494756    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.494846    4877 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.494880    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.494905    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.494927    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.494906792 +0000 UTC m=+16.923796192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : configmap "kube-rbac-proxy" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.494979    4877 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.495005    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495030    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs podName:060b1735-396e-43d2-940d-e5b9bd3ec06b nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.495015513 +0000 UTC m=+16.923904923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs") pod "network-metrics-daemon-z74kk" (UID: "060b1735-396e-43d2-940d-e5b9bd3ec06b") : secret "metrics-daemon-secret" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.495066    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495077    4877 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.495105    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495114    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert podName:ba135d3a-8995-4181-a315-51ae998bba50 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.495103074 +0000 UTC m=+16.923992476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lrntf" (UID: "ba135d3a-8995-4181-a315-51ae998bba50") : secret "networking-console-plugin-cert" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.495142    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495167    4877 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495208    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls podName:bec84fda-07e9-469d-bec4-16e9586fed14 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.495195911 +0000 UTC m=+16.924085321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c5v7s" (UID: "bec84fda-07e9-469d-bec4-16e9586fed14") : secret "samples-operator-tls" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495225    4877 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495239    4877 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495277    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls podName:9c1080c7-2fbb-471b-9ea9-b05fc60f95e9 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.495267636 +0000 UTC m=+16.924157032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls") pod "dns-default-cfphv" (UID: "9c1080c7-2fbb-471b-9ea9-b05fc60f95e9") : secret "dns-default-metrics-tls" not found
Oct 30 03:03:00.495352 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:00.495293    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls podName:a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e nodeName:}" failed. No retries permitted until 2026-10-30 03:03:08.495285276 +0000 UTC m=+16.924174672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xq4n" (UID: "a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e") : secret "insights-runtime-extractor-tls" not found
Oct 30 03:03:00.497171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.497150    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"image-registry-7cd96bbd68-84shk\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") " pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:00.497363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.497342    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74e50c74-23f8-46b7-98e6-6ac103daba15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ph5d9\" (UID: \"74e50c74-23f8-46b7-98e6-6ac103daba15\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:03:00.497408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.497347    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"image-registry-7bf5b6fbbf-6hq7m\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") " pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:00.502088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.502069    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"
Oct 30 03:03:00.504246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.504228    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:00.506752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.506738    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:00.813259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.813230    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9"]
Oct 30 03:03:00.814544 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:03:00.814521    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e50c74_23f8_46b7_98e6_6ac103daba15.slice/crio-111eed52b845b1cbb67a42a045ff4f5a9d67e64b46b4b7c5fa8494e6692ed35d WatchSource:0}: Error finding container 111eed52b845b1cbb67a42a045ff4f5a9d67e64b46b4b7c5fa8494e6692ed35d: Status 404 returned error can't find the container with id 111eed52b845b1cbb67a42a045ff4f5a9d67e64b46b4b7c5fa8494e6692ed35d
Oct 30 03:03:00.816527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.816509    4877 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:03:00.825602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.825584    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd96bbd68-84shk"]
Oct 30 03:03:00.834894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.834874    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"]
Oct 30 03:03:00.835519 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:03:00.835500    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcca0b47_f1cc_4b6f_b20d_ebfb450be528.slice/crio-014a96bcdd54d162d0d3ef0de7d55f737fa1ea2fc46b3599124e3bcbdb332878 WatchSource:0}: Error finding container 014a96bcdd54d162d0d3ef0de7d55f737fa1ea2fc46b3599124e3bcbdb332878: Status 404 returned error can't find the container with id 014a96bcdd54d162d0d3ef0de7d55f737fa1ea2fc46b3599124e3bcbdb332878
Oct 30 03:03:00.878614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.878595    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:03:00.878704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.878686    4877 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:03:00.880260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:00.880242    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59ddf86b9f-2wfp9"
Oct 30 03:03:01.773749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:01.773709    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk" event={"ID":"2d8497ef-8d61-4e71-924e-ed835a093f9d","Type":"ContainerStarted","Data":"10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8"}
Oct 30 03:03:01.774231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:01.773769    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk" event={"ID":"2d8497ef-8d61-4e71-924e-ed835a093f9d","Type":"ContainerStarted","Data":"52f0c25a65bd6623dcce8e0aca15c3b8135ad3895bc87cb4c09d3be229ccdb5a"}
Oct 30 03:03:01.775418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:01.775390    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m" event={"ID":"fcca0b47-f1cc-4b6f-b20d-ebfb450be528","Type":"ContainerStarted","Data":"fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39"}
Oct 30 03:03:01.775547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:01.775450    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m" event={"ID":"fcca0b47-f1cc-4b6f-b20d-ebfb450be528","Type":"ContainerStarted","Data":"014a96bcdd54d162d0d3ef0de7d55f737fa1ea2fc46b3599124e3bcbdb332878"}
Oct 30 03:03:01.776749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:01.776725    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9" event={"ID":"74e50c74-23f8-46b7-98e6-6ac103daba15","Type":"ContainerStarted","Data":"111eed52b845b1cbb67a42a045ff4f5a9d67e64b46b4b7c5fa8494e6692ed35d"}
Oct 30 03:03:02.781528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:02.781491    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ph5d9" event={"ID":"74e50c74-23f8-46b7-98e6-6ac103daba15","Type":"ContainerStarted","Data":"05597c61dca5efc49242df37c3990ff2ebbd7b8757309a8554f127e380316097"}
Oct 30 03:03:02.781528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:02.781530    4877 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:03:02.782066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:02.781745    4877 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 03:03:02.962142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:02.962110    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:04.690310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:04.690272    4877 scope.go:122] "RemoveContainer" containerID="3ae9fe9f7bc13df9ae35a77cd77bab5625c123e57db7df5eeb0ae662675c8a7a"
Oct 30 03:03:04.790364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:04.790339    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:03:04.790511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:04.790412    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" event={"ID":"3a65983f-48e3-4e8f-a576-2110dfbb2ec2","Type":"ContainerStarted","Data":"d330fee069b847740c08fb17d8284b6f8f537055f80be791c7c398e200c8dd71"}
Oct 30 03:03:04.790710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:04.790692    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:03:04.791286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:04.791263    4877 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-qjtcv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body=
Oct 30 03:03:04.791395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:04.791300    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv" podUID="3a65983f-48e3-4e8f-a576-2110dfbb2ec2" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.11:8443/readyz\": dial tcp 10.133.0.11:8443: connect: connection refused"
Oct 30 03:03:05.797911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:05.797874    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qjtcv"
Oct 30 03:03:07.977079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:07.977044    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:08.563700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.563667    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:03:08.563700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.563699    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:03:08.563922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.563720    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:08.563922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.563751    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:03:08.563922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.563877    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:03:08.564094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.563929    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:08.564385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.564352    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:08.566310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.566284    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xq4n\" (UID: \"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e\") " pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:08.566310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.566302    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba135d3a-8995-4181-a315-51ae998bba50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lrntf\" (UID: \"ba135d3a-8995-4181-a315-51ae998bba50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:03:08.566476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.566457    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bec84fda-07e9-469d-bec4-16e9586fed14-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c5v7s\" (UID: \"bec84fda-07e9-469d-bec4-16e9586fed14\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:03:08.566526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.566511    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060b1735-396e-43d2-940d-e5b9bd3ec06b-metrics-certs\") pod \"network-metrics-daemon-z74kk\" (UID: \"060b1735-396e-43d2-940d-e5b9bd3ec06b\") " pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:03:08.566559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.566511    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c1080c7-2fbb-471b-9ea9-b05fc60f95e9-metrics-tls\") pod \"dns-default-cfphv\" (UID: \"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9\") " pod="openshift-dns/dns-default-cfphv"
Oct 30 03:03:08.603486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.603462    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"
Oct 30 03:03:08.605585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.605566    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf"
Oct 30 03:03:08.606651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.606636    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z74kk"
Oct 30 03:03:08.610854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.610837    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cfphv"
Oct 30 03:03:08.614290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.614268    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xq4n"
Oct 30 03:03:08.964287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.964258    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s"]
Oct 30 03:03:08.982241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.982165    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cfphv"]
Oct 30 03:03:08.983683 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:03:08.983567    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba135d3a_8995_4181_a315_51ae998bba50.slice/crio-a704143501b9627ae070da9fca9e33985259b5771d28d0329b87aa73749ed8e6 WatchSource:0}: Error finding container a704143501b9627ae070da9fca9e33985259b5771d28d0329b87aa73749ed8e6: Status 404 returned error can't find the container with id a704143501b9627ae070da9fca9e33985259b5771d28d0329b87aa73749ed8e6
Oct 30 03:03:08.984631 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:03:08.984604    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1080c7_2fbb_471b_9ea9_b05fc60f95e9.slice/crio-8a7f9dbcee147735455ed8a6a0e3da64c9c6f1c580b6c63056dd5a896b045e46 WatchSource:0}: Error finding container 8a7f9dbcee147735455ed8a6a0e3da64c9c6f1c580b6c63056dd5a896b045e46: Status 404 returned error can't find the container with id 8a7f9dbcee147735455ed8a6a0e3da64c9c6f1c580b6c63056dd5a896b045e46
Oct 30 03:03:08.987544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.987476    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lrntf"]
Oct 30 03:03:08.994132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.994111    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z74kk"]
Oct 30 03:03:08.994889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:08.994868    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9xq4n"]
Oct 30 03:03:09.003285 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:03:09.003264    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e5c46c_1ff0_4cb4_8ac7_77a160f3bd6e.slice/crio-bc0ceaf87c26f41273199f5b57dceaa9d2d415799e1e3e94b79c0053e55935b3 WatchSource:0}: Error finding container bc0ceaf87c26f41273199f5b57dceaa9d2d415799e1e3e94b79c0053e55935b3: Status 404 returned error can't find the container with id bc0ceaf87c26f41273199f5b57dceaa9d2d415799e1e3e94b79c0053e55935b3
Oct 30 03:03:09.038090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.038065    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"]
Oct 30 03:03:09.810661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.810544    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xq4n" event={"ID":"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e","Type":"ContainerStarted","Data":"1ce26ba85ceb744d0655a9659821bd6b0c75ac904f7b591e194fb9f7fe49ddaf"}
Oct 30 03:03:09.810661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.810609    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xq4n" event={"ID":"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e","Type":"ContainerStarted","Data":"bc0ceaf87c26f41273199f5b57dceaa9d2d415799e1e3e94b79c0053e55935b3"}
Oct 30 03:03:09.813529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.813459    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z74kk" event={"ID":"060b1735-396e-43d2-940d-e5b9bd3ec06b","Type":"ContainerStarted","Data":"7a899aaacd267650eb24691d9ad67f693cea4a29fee8c37849f19f25f20784f9"}
Oct 30 03:03:09.817376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.817342    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cfphv" event={"ID":"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9","Type":"ContainerStarted","Data":"8a7f9dbcee147735455ed8a6a0e3da64c9c6f1c580b6c63056dd5a896b045e46"}
Oct 30 03:03:09.819471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.819443    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s" event={"ID":"bec84fda-07e9-469d-bec4-16e9586fed14","Type":"ContainerStarted","Data":"d4268c1c942075767758feb9c675f4b2e8facf83be2b47664a838dc883a24d0f"}
Oct 30 03:03:09.821311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:09.821254    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf" event={"ID":"ba135d3a-8995-4181-a315-51ae998bba50","Type":"ContainerStarted","Data":"a704143501b9627ae070da9fca9e33985259b5771d28d0329b87aa73749ed8e6"}
Oct 30 03:03:10.828260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:10.828226    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lrntf" event={"ID":"ba135d3a-8995-4181-a315-51ae998bba50","Type":"ContainerStarted","Data":"a48ddae1db49a66edd08bd212a9f595ce5c4782ebeff4e88d1e3caceff042128"}
Oct 30 03:03:10.837509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:10.837481    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xq4n" event={"ID":"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e","Type":"ContainerStarted","Data":"46abbce9939a8561c07e0acd389f0aaf16b0660f71c899e37505c71adc615303"}
Oct 30 03:03:10.845179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:10.845151    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z74kk" event={"ID":"060b1735-396e-43d2-940d-e5b9bd3ec06b","Type":"ContainerStarted","Data":"e06858dac2f99fa47145d918be1a826bfd9de414baf09a8349b87e1654490125"}
Oct 30 03:03:10.845295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:10.845207    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z74kk" event={"ID":"060b1735-396e-43d2-940d-e5b9bd3ec06b","Type":"ContainerStarted","Data":"24d21ba8fafcfad00f321e8ad25f661179029af67a064b2ef844922d7bdb4d55"}
Oct 30 03:03:11.416111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.416087    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sfv5t"
Oct 30 03:03:11.527801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.527772    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z54vn"
Oct 30 03:03:11.850168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.850094    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cfphv" event={"ID":"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9","Type":"ContainerStarted","Data":"ec087f42b56d9bf288481a9cedd3b8474845d029042c5c359c138d47b0110698"}
Oct 30 03:03:11.850168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.850143    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cfphv" event={"ID":"9c1080c7-2fbb-471b-9ea9-b05fc60f95e9","Type":"ContainerStarted","Data":"f3dc9a48d27518bfa19bafb63c844cc941b322c3314e8f7250611af25a4e233d"}
Oct 30 03:03:11.850652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.850225    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cfphv"
Oct 30 03:03:11.851979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.851950    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s" event={"ID":"bec84fda-07e9-469d-bec4-16e9586fed14","Type":"ContainerStarted","Data":"42087f04ac9ff9d14a93d9576d7ba814f5c5b142849de1e85809c5ab9ef7ea3c"}
Oct 30 03:03:11.852098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.851999    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c5v7s" event={"ID":"bec84fda-07e9-469d-bec4-16e9586fed14","Type":"ContainerStarted","Data":"e2f6f68a528dbc47e3151e39db53701ee2f6111980f2f07946fbd6b6b53e2f07"}
Oct 30 03:03:11.854010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:11.853983    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xq4n" event={"ID":"a7e5c46c-1ff0-4cb4-8ac7-77a160f3bd6e","Type":"ContainerStarted","Data":"7d4d4cd9be7e1d8cdc032e6bda4dbc74c99dc830854461b9389ff0759d436843"}
Oct 30 03:03:16.227157 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:16.227111    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:03:17.771401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.771367    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g26vv"]
Oct 30 03:03:17.807648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.807619    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834633    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26cf120d-2eae-4536-9da5-c936e7f9b17a-metrics-client-ca\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834665    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834690    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-sys\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834791    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834817    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwk7\" (UniqueName: \"kubernetes.io/projected/26cf120d-2eae-4536-9da5-c936e7f9b17a-kube-api-access-hnwk7\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834834    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-wtmp\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.834906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834895    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-textfile\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.835111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.834975    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-root\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.835111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.835014    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-tls\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.862042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.862018    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 03:03:17.862205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.862181    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 03:03:17.862411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.862388    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5lzhh\""
Oct 30 03:03:17.862533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.862413    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 03:03:17.862533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.862475    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 03:03:17.935525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935496    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26cf120d-2eae-4536-9da5-c936e7f9b17a-metrics-client-ca\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935530    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935551    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-sys\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935578    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935601    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwk7\" (UniqueName: \"kubernetes.io/projected/26cf120d-2eae-4536-9da5-c936e7f9b17a-kube-api-access-hnwk7\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935628    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-wtmp\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935646    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-sys\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935669    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-textfile\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935699    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-root\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935741    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-tls\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935791    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-wtmp\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.935990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.935836    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26cf120d-2eae-4536-9da5-c936e7f9b17a-root\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.936150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.936016    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-textfile\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.936150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.936112    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26cf120d-2eae-4536-9da5-c936e7f9b17a-metrics-client-ca\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.936248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.936181    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-accelerators-collector-config\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.937932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.937910    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-tls\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.938025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.938011    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26cf120d-2eae-4536-9da5-c936e7f9b17a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:17.957339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:17.957313    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwk7\" (UniqueName: \"kubernetes.io/projected/26cf120d-2eae-4536-9da5-c936e7f9b17a-kube-api-access-hnwk7\") pod \"node-exporter-g26vv\" (UID: \"26cf120d-2eae-4536-9da5-c936e7f9b17a\") " pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:18.156694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:18.156630    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g26vv"
Oct 30 03:03:18.195276 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:03:18.195244    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf120d_2eae_4536_9da5_c936e7f9b17a.slice/crio-591272bf78ddd20197bc8e1bd93ab8f36ec251f8809d608c486bf2648c179814 WatchSource:0}: Error finding container 591272bf78ddd20197bc8e1bd93ab8f36ec251f8809d608c486bf2648c179814: Status 404 returned error can't find the container with id 591272bf78ddd20197bc8e1bd93ab8f36ec251f8809d608c486bf2648c179814
Oct 30 03:03:18.883113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:18.883063    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g26vv" event={"ID":"26cf120d-2eae-4536-9da5-c936e7f9b17a","Type":"ContainerStarted","Data":"591272bf78ddd20197bc8e1bd93ab8f36ec251f8809d608c486bf2648c179814"}
Oct 30 03:03:19.044417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:19.044378    4877 patch_prober.go:29] interesting pod/image-registry-7bf5b6fbbf-6hq7m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 03:03:19.044603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:19.044459    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m" podUID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:03:19.657529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:19.657504    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6ssh4"
Oct 30 03:03:19.888565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:19.888483    4877 generic.go:356] "Generic (PLEG): container finished" podID="26cf120d-2eae-4536-9da5-c936e7f9b17a" containerID="c48a3ed8a1d0fb9aff49c07fae7804ea30fc17cd330facdf31d8d376957c5d6c" exitCode=0
Oct 30 03:03:19.888899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:19.888574    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g26vv" event={"ID":"26cf120d-2eae-4536-9da5-c936e7f9b17a","Type":"ContainerDied","Data":"c48a3ed8a1d0fb9aff49c07fae7804ea30fc17cd330facdf31d8d376957c5d6c"}
Oct 30 03:03:20.621989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:20.621953    4877 patch_prober.go:29] interesting pod/image-registry-7cd96bbd68-84shk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 03:03:20.622167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:20.622004    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk" podUID="2d8497ef-8d61-4e71-924e-ed835a093f9d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:03:20.895464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:20.895374    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g26vv" event={"ID":"26cf120d-2eae-4536-9da5-c936e7f9b17a","Type":"ContainerStarted","Data":"6c364d59a840b41d9281f81541be42270b917bead1552b121910919056e5c389"}
Oct 30 03:03:20.895464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:20.895441    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g26vv" event={"ID":"26cf120d-2eae-4536-9da5-c936e7f9b17a","Type":"ContainerStarted","Data":"7f5c9b38900eda53b697d445afd212b06b22a80aea12d5aad017cea36794d23f"}
Oct 30 03:03:20.949136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:20.949088    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g26vv" podStartSLOduration=2.461073281 podStartE2EDuration="3.949076147s" totalImagesPullingTime="1.406186421s" totalInitContainerRuntime="81.816445ms" isStatefulPod=true podCreationTimestamp="2026-10-30 03:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:20.9484938 +0000 UTC m=+29.377383219" watchObservedRunningTime="2026-10-30 03:03:20.949076147 +0000 UTC m=+29.377965561"
Oct 30 03:03:21.859977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:21.859946    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cfphv"
Oct 30 03:03:23.788628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:23.788594    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:29.042248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:29.042217    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:31.287007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:31.286978    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd96bbd68-84shk"]
Oct 30 03:03:34.058042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.057924    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m" podUID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" containerName="registry" containerID="cri-o://fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39" gracePeriod=30
Oct 30 03:03:34.289997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.289969    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:34.353579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353507    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353549    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353716    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353760    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353800    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353847    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353880    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.353919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.353900    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") pod \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\" (UID: \"fcca0b47-f1cc-4b6f-b20d-ebfb450be528\") "
Oct 30 03:03:34.354645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.354536    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:03:34.355720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.355682    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:03:34.356226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.356180    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:03:34.356312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.356262    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:03:34.356888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.356860    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:03:34.357043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.357015    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "kube-api-access-mdsfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:03:34.357163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.357142    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:03:34.362317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.362292    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted" pod "fcca0b47-f1cc-4b6f-b20d-ebfb450be528" (UID: "fcca0b47-f1cc-4b6f-b20d-ebfb450be528"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:03:34.454568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454544    4877 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-ca-trust-extracted\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454566    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdsfn\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-kube-api-access-mdsfn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454577    4877 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-bound-sa-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454586    4877 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-trusted-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454595    4877 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-certificates\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454603    4877 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-registry-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454612    4877 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-installation-pull-secrets\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.454704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.454621    4877 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcca0b47-f1cc-4b6f-b20d-ebfb450be528-image-registry-private-configuration\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:34.949048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.949005    4877 generic.go:356] "Generic (PLEG): container finished" podID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" containerID="fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39" exitCode=0
Oct 30 03:03:34.949048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.949048    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"
Oct 30 03:03:34.949266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.949094    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m" event={"ID":"fcca0b47-f1cc-4b6f-b20d-ebfb450be528","Type":"ContainerDied","Data":"fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39"}
Oct 30 03:03:34.949266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.949145    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m" event={"ID":"fcca0b47-f1cc-4b6f-b20d-ebfb450be528","Type":"ContainerDied","Data":"014a96bcdd54d162d0d3ef0de7d55f737fa1ea2fc46b3599124e3bcbdb332878"}
Oct 30 03:03:34.949266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.949167    4877 scope.go:122] "RemoveContainer" containerID="fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39"
Oct 30 03:03:34.957571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.957547    4877 scope.go:122] "RemoveContainer" containerID="fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39"
Oct 30 03:03:34.957875 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:34.957854    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39\": container with ID starting with fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39 not found: ID does not exist" containerID="fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39"
Oct 30 03:03:34.957918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:34.957884    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39"} err="failed to get container status \"fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39\": rpc error: code = NotFound desc = could not find container \"fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39\": container with ID starting with fd8237cc79b48bc120f35702256ab917297b9886b45f76ca8c640058e6fe2f39 not found: ID does not exist"
Oct 30 03:03:35.012263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:35.012233    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"]
Oct 30 03:03:35.024194 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:35.024168    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bf5b6fbbf-6hq7m"]
Oct 30 03:03:35.695419 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:35.695389    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" path="/var/lib/kubelet/pods/fcca0b47-f1cc-4b6f-b20d-ebfb450be528/volumes"
Oct 30 03:03:46.227831 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:46.227790    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:03:56.306722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.306662    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk" podUID="2d8497ef-8d61-4e71-924e-ed835a093f9d" containerName="registry" containerID="cri-o://10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8" gracePeriod=30
Oct 30 03:03:56.553915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.553883    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:56.607815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607757    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.607815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607808    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.607970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607836    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.607970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607870    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.607970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607890    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.607970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607908    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.607970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607931    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.608227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.607995    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") pod \"2d8497ef-8d61-4e71-924e-ed835a093f9d\" (UID: \"2d8497ef-8d61-4e71-924e-ed835a093f9d\") "
Oct 30 03:03:56.608359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.608327    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:03:56.608471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.608348    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:03:56.612236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.612034    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:03:56.612545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.612518    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "kube-api-access-zbdb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:03:56.612662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.612643    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:03:56.612961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.612817    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:03:56.613606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.613576    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:03:56.616664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.616641    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted" pod "2d8497ef-8d61-4e71-924e-ed835a093f9d" (UID: "2d8497ef-8d61-4e71-924e-ed835a093f9d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:03:56.709138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709106    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbdb8\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-kube-api-access-zbdb8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709135    4877 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-tls\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709145    4877 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d8497ef-8d61-4e71-924e-ed835a093f9d-ca-trust-extracted\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709157    4877 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-image-registry-private-configuration\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709167    4877 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-registry-certificates\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709175    4877 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d8497ef-8d61-4e71-924e-ed835a093f9d-trusted-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709184    4877 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d8497ef-8d61-4e71-924e-ed835a093f9d-installation-pull-secrets\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:56.709320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:56.709193    4877 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d8497ef-8d61-4e71-924e-ed835a093f9d-bound-sa-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:03:57.040153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.040112    4877 generic.go:356] "Generic (PLEG): container finished" podID="2d8497ef-8d61-4e71-924e-ed835a093f9d" containerID="10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8" exitCode=0
Oct 30 03:03:57.040322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.040175    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk"
Oct 30 03:03:57.040322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.040200    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk" event={"ID":"2d8497ef-8d61-4e71-924e-ed835a093f9d","Type":"ContainerDied","Data":"10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8"}
Oct 30 03:03:57.040322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.040275    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd96bbd68-84shk" event={"ID":"2d8497ef-8d61-4e71-924e-ed835a093f9d","Type":"ContainerDied","Data":"52f0c25a65bd6623dcce8e0aca15c3b8135ad3895bc87cb4c09d3be229ccdb5a"}
Oct 30 03:03:57.040322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.040296    4877 scope.go:122] "RemoveContainer" containerID="10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8"
Oct 30 03:03:57.049306 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.049289    4877 scope.go:122] "RemoveContainer" containerID="10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8"
Oct 30 03:03:57.049644 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:03:57.049626    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8\": container with ID starting with 10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8 not found: ID does not exist" containerID="10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8"
Oct 30 03:03:57.049691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.049652    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8"} err="failed to get container status \"10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8\": rpc error: code = NotFound desc = could not find container \"10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8\": container with ID starting with 10958c9ea406ebb3eac368b760861e2a7604bb9e2d174792066ed9b7a899ece8 not found: ID does not exist"
Oct 30 03:03:57.105581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.105553    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd96bbd68-84shk"]
Oct 30 03:03:57.117776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.117754    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cd96bbd68-84shk"]
Oct 30 03:03:57.695004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:03:57.694974    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8497ef-8d61-4e71-924e-ed835a093f9d" path="/var/lib/kubelet/pods/2d8497ef-8d61-4e71-924e-ed835a093f9d/volumes"
Oct 30 03:04:12.092739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.092709    4877 generic.go:356] "Generic (PLEG): container finished" podID="3e49f31c-2187-47b3-89d9-1e76ec6f883a" containerID="e4cf8d9f84dcd4c245051bfdbec4bd0964b4e14033ce053f96bb86a2e8a4194a" exitCode=0
Oct 30 03:04:12.093126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.092785    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-klw22" event={"ID":"3e49f31c-2187-47b3-89d9-1e76ec6f883a","Type":"ContainerDied","Data":"e4cf8d9f84dcd4c245051bfdbec4bd0964b4e14033ce053f96bb86a2e8a4194a"}
Oct 30 03:04:12.093126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.093100    4877 scope.go:122] "RemoveContainer" containerID="e4cf8d9f84dcd4c245051bfdbec4bd0964b4e14033ce053f96bb86a2e8a4194a"
Oct 30 03:04:12.950905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.950873    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sdpwc-wvr85"]
Oct 30 03:04:12.951157 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.951145    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2d8497ef-8d61-4e71-924e-ed835a093f9d" containerName="registry"
Oct 30 03:04:12.951211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.951158    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8497ef-8d61-4e71-924e-ed835a093f9d" containerName="registry"
Oct 30 03:04:12.951211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.951177    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" containerName="registry"
Oct 30 03:04:12.951211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.951183    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" containerName="registry"
Oct 30 03:04:12.951316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.951233    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcca0b47-f1cc-4b6f-b20d-ebfb450be528" containerName="registry"
Oct 30 03:04:12.951316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.951244    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8497ef-8d61-4e71-924e-ed835a093f9d" containerName="registry"
Oct 30 03:04:12.955544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.955520    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:12.976214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:12.976190    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sdpwc-wvr85"]
Oct 30 03:04:13.037600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.037574    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.037710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.037602    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.037710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.037651    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254ss\" (UniqueName: \"kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.037800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.037721    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.097255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.097223    4877 generic.go:356] "Generic (PLEG): container finished" podID="f6a644f9-7b06-4072-8052-704244dd5a8d" containerID="774e9e0339d9c2fd8620171bfd12176d4cdebbc215cdc9fff7cb7c124421f1cd" exitCode=0
Oct 30 03:04:13.097646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.097297    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4" event={"ID":"f6a644f9-7b06-4072-8052-704244dd5a8d","Type":"ContainerDied","Data":"774e9e0339d9c2fd8620171bfd12176d4cdebbc215cdc9fff7cb7c124421f1cd"}
Oct 30 03:04:13.097646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.097636    4877 scope.go:122] "RemoveContainer" containerID="774e9e0339d9c2fd8620171bfd12176d4cdebbc215cdc9fff7cb7c124421f1cd"
Oct 30 03:04:13.099277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.099248    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-klw22" event={"ID":"3e49f31c-2187-47b3-89d9-1e76ec6f883a","Type":"ContainerStarted","Data":"ae9509a39fd63fd834e132fde15dab4cd1d4efb8ddef809d41553361f4cd5850"}
Oct 30 03:04:13.138345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.138320    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.138455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.138378    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.138542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.138525    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.138617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.138600    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254ss\" (UniqueName: \"kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.138801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.138782    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.139083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.139062    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.139197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.139179    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.159055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.159028    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254ss\" (UniqueName: \"kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss\") pod \"periodic-gathering-sdpwc-wvr85\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") " pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.266662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.266582    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:04:13.562273 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:13.562249    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sdpwc-wvr85"]
Oct 30 03:04:13.563501 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:04:13.563476    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289349d9_cd56_4516_8a2f_b701ff4b7a08.slice/crio-5063ab598ba9f67456c051b947bdd5e10a622190d1bad4f2620ab704c3451f14 WatchSource:0}: Error finding container 5063ab598ba9f67456c051b947bdd5e10a622190d1bad4f2620ab704c3451f14: Status 404 returned error can't find the container with id 5063ab598ba9f67456c051b947bdd5e10a622190d1bad4f2620ab704c3451f14
Oct 30 03:04:14.104018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:14.103982    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-g6lv4" event={"ID":"f6a644f9-7b06-4072-8052-704244dd5a8d","Type":"ContainerStarted","Data":"2eb7f1b5eafa34b144ca34420ac29098d2f13bde0b2d3ac027799ce81738de64"}
Oct 30 03:04:14.105491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:14.105460    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdpwc-wvr85" event={"ID":"289349d9-cd56-4516-8a2f-b701ff4b7a08","Type":"ContainerStarted","Data":"de07f1bb35667e43160fa513a18ffa7fc3d9ccb073a5e436e4bebaec2199da3f"}
Oct 30 03:04:14.105632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:14.105505    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdpwc-wvr85" event={"ID":"289349d9-cd56-4516-8a2f-b701ff4b7a08","Type":"ContainerStarted","Data":"5063ab598ba9f67456c051b947bdd5e10a622190d1bad4f2620ab704c3451f14"}
Oct 30 03:04:14.182379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:14.182319    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sdpwc-wvr85" podStartSLOduration=2.182302194 podStartE2EDuration="2.182302194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:14.1819217 +0000 UTC m=+82.610811155" watchObservedRunningTime="2026-10-30 03:04:14.182302194 +0000 UTC m=+82.611191611"
Oct 30 03:04:16.227144 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:04:16.227107    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:04:23.137444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:23.137390    4877 generic.go:356] "Generic (PLEG): container finished" podID="1e29e28e-149b-4e87-af1c-a1ab78aa1969" containerID="937c65182e08075fdc82fdab0cdd8f5c7685e6a1db6136e7033db148d0727019" exitCode=0
Oct 30 03:04:23.137847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:23.137467    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk" event={"ID":"1e29e28e-149b-4e87-af1c-a1ab78aa1969","Type":"ContainerDied","Data":"937c65182e08075fdc82fdab0cdd8f5c7685e6a1db6136e7033db148d0727019"}
Oct 30 03:04:23.137847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:23.137757    4877 scope.go:122] "RemoveContainer" containerID="937c65182e08075fdc82fdab0cdd8f5c7685e6a1db6136e7033db148d0727019"
Oct 30 03:04:24.142936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:04:24.142898    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-plhvk" event={"ID":"1e29e28e-149b-4e87-af1c-a1ab78aa1969","Type":"ContainerStarted","Data":"54f8354aa142d2baa2554617c6bac09aa154b566175e47a9766adfa6b207cb61"}
Oct 30 03:04:46.227263 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:04:46.227222    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:05:16.227623 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:05:16.227544    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:05:37.397928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:37.397843    4877 generic.go:356] "Generic (PLEG): container finished" podID="289349d9-cd56-4516-8a2f-b701ff4b7a08" containerID="de07f1bb35667e43160fa513a18ffa7fc3d9ccb073a5e436e4bebaec2199da3f" exitCode=0
Oct 30 03:05:37.397928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:37.397889    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdpwc-wvr85" event={"ID":"289349d9-cd56-4516-8a2f-b701ff4b7a08","Type":"ContainerDied","Data":"de07f1bb35667e43160fa513a18ffa7fc3d9ccb073a5e436e4bebaec2199da3f"}
Oct 30 03:05:38.526106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.526079    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:05:38.679782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.679674    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle\") pod \"289349d9-cd56-4516-8a2f-b701ff4b7a08\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") "
Oct 30 03:05:38.679782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.679758    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss\" (UniqueName: \"kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss\") pod \"289349d9-cd56-4516-8a2f-b701ff4b7a08\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") "
Oct 30 03:05:38.680015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.679825    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path\") pod \"289349d9-cd56-4516-8a2f-b701ff4b7a08\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") "
Oct 30 03:05:38.680015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.679850    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle\") pod \"289349d9-cd56-4516-8a2f-b701ff4b7a08\" (UID: \"289349d9-cd56-4516-8a2f-b701ff4b7a08\") "
Oct 30 03:05:38.680121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.680097    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle" pod "289349d9-cd56-4516-8a2f-b701ff4b7a08" (UID: "289349d9-cd56-4516-8a2f-b701ff4b7a08"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:38.680243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.680215    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path" pod "289349d9-cd56-4516-8a2f-b701ff4b7a08" (UID: "289349d9-cd56-4516-8a2f-b701ff4b7a08"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:05:38.680304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.680288    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle" pod "289349d9-cd56-4516-8a2f-b701ff4b7a08" (UID: "289349d9-cd56-4516-8a2f-b701ff4b7a08"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:05:38.682011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.681994    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss" pod "289349d9-cd56-4516-8a2f-b701ff4b7a08" (UID: "289349d9-cd56-4516-8a2f-b701ff4b7a08"). InnerVolumeSpecName "kube-api-access-254ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:05:38.780968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.780931    4877 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-service-ca-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:05:38.780968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.780961    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254ss\" (UniqueName: \"kubernetes.io/projected/289349d9-cd56-4516-8a2f-b701ff4b7a08-kube-api-access-254ss\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:05:38.780968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.780971    4877 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/289349d9-cd56-4516-8a2f-b701ff4b7a08-archives-path\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:05:38.780968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:38.780980    4877 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289349d9-cd56-4516-8a2f-b701ff4b7a08-trusted-ca-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:05:39.405369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:39.405330    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdpwc-wvr85" event={"ID":"289349d9-cd56-4516-8a2f-b701ff4b7a08","Type":"ContainerDied","Data":"5063ab598ba9f67456c051b947bdd5e10a622190d1bad4f2620ab704c3451f14"}
Oct 30 03:05:39.405557 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:39.405381    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5063ab598ba9f67456c051b947bdd5e10a622190d1bad4f2620ab704c3451f14"
Oct 30 03:05:39.405557 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:05:39.405347    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdpwc-wvr85"
Oct 30 03:05:46.227378 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:05:46.227335    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:06:16.227588 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:06:16.227547    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:06:28.089002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.088968    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"]
Oct 30 03:06:28.089492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.089267    4877 cpu_manager.go:414] "containerMap: removing container" podUID="289349d9-cd56-4516-8a2f-b701ff4b7a08" containerName="insights-gathering"
Oct 30 03:06:28.089492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.089279    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289349d9-cd56-4516-8a2f-b701ff4b7a08" containerName="insights-gathering"
Oct 30 03:06:28.089492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.089353    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="289349d9-cd56-4516-8a2f-b701ff4b7a08" containerName="insights-gathering"
Oct 30 03:06:28.092442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.092414    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.119258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.119227    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-snljv\""
Oct 30 03:06:28.119527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.119487    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 03:06:28.119527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.119488    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 03:06:28.120546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.120517    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"]
Oct 30 03:06:28.220507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.220479    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.220643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.220514    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zdn\" (UniqueName: \"kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.220643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.220613    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.321283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.321253    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zdn\" (UniqueName: \"kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.321407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.321310    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.321407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.321339    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.321663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.321648    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.321745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.321726    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.340929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.340884    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zdn\" (UniqueName: \"kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.408508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.408481    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:06:28.701523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:28.701492    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"]
Oct 30 03:06:28.702804 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:28.702778    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f6e3e3_717c_4bff_80e2_7039cad092ea.slice/crio-85c37842980cfcd83cc21c5b03ecfeaa03533b347c92120cba03574bbebe668d WatchSource:0}: Error finding container 85c37842980cfcd83cc21c5b03ecfeaa03533b347c92120cba03574bbebe668d: Status 404 returned error can't find the container with id 85c37842980cfcd83cc21c5b03ecfeaa03533b347c92120cba03574bbebe668d
Oct 30 03:06:29.296741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.296708    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"]
Oct 30 03:06:29.300204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.300183    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.327695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.327650    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-w5dhh\""
Oct 30 03:06:29.329383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.329356    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"]
Oct 30 03:06:29.393783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.393740    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"]
Oct 30 03:06:29.397276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.397256    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.411743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.411720    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5rbxg\""
Oct 30 03:06:29.416725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.416705    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"]
Oct 30 03:06:29.429801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.429774    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq2l\" (UniqueName: \"kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.429948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.429831    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.429948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.429894    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.429948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.429914    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.430114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.430016    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vrf\" (UniqueName: \"kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.430114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.430092    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.501612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.501576    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"]
Oct 30 03:06:29.505326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.505305    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.519566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.519544    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xg57q\""
Oct 30 03:06:29.524268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.524208    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"]
Oct 30 03:06:29.531097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531076    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq2l\" (UniqueName: \"kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.531217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531120    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.531217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531141    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.531217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531156    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.531217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531175    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vrf\" (UniqueName: \"kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.531217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531209    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.531598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.531579    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.532090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.532066    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.532324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.532307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.532561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.532542    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.550755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.550678    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq2l\" (UniqueName: \"kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.551268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.551243    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vrf\" (UniqueName: \"kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.580086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.580056    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l" event={"ID":"70f6e3e3-717c-4bff-80e2-7039cad092ea","Type":"ContainerStarted","Data":"85c37842980cfcd83cc21c5b03ecfeaa03533b347c92120cba03574bbebe668d"}
Oct 30 03:06:29.597873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.597844    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"]
Oct 30 03:06:29.602009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.601990    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.616505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.616485    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-b8s4s\""
Oct 30 03:06:29.619708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.619685    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:06:29.621578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.621557    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"]
Oct 30 03:06:29.632069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.632034    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.632166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.632095    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.632231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.632166    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm45\" (UniqueName: \"kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.715945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.715908    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:06:29.733263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.733229    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.733380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.733295    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm45\" (UniqueName: \"kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.733380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.733339    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.733380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.733371    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.733812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.733792    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.734010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.733978    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.734101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.734026    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.734101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.734070    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttth\" (UniqueName: \"kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.753599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.753564    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm45\" (UniqueName: \"kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.823896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.823819    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:06:29.835084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.835056    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.835178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.835105    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.835178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.835144    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttth\" (UniqueName: \"kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.835588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.835562    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.835702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.835570    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.854867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.854840    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttth\" (UniqueName: \"kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.920773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.920744    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:06:29.925528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:29.925497    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"]
Oct 30 03:06:29.926133 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:29.926104    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011367c3_c75c_4875_bbe4_46ac08eb8382.slice/crio-35474418dba074febbb854528d413fc8f981c3e548641c5221f7ef67df4129b5 WatchSource:0}: Error finding container 35474418dba074febbb854528d413fc8f981c3e548641c5221f7ef67df4129b5: Status 404 returned error can't find the container with id 35474418dba074febbb854528d413fc8f981c3e548641c5221f7ef67df4129b5
Oct 30 03:06:30.019386 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.019356    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"]
Oct 30 03:06:30.020319 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:30.020291    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70c2e90_1d7e_46eb_b922_f40c0056f43d.slice/crio-b1630ea507d7a8e88457a949985a73785d0762f7ffb0b62d8b32911dc7810fd6 WatchSource:0}: Error finding container b1630ea507d7a8e88457a949985a73785d0762f7ffb0b62d8b32911dc7810fd6: Status 404 returned error can't find the container with id b1630ea507d7a8e88457a949985a73785d0762f7ffb0b62d8b32911dc7810fd6
Oct 30 03:06:30.131796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.131768    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"]
Oct 30 03:06:30.132520 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:30.132490    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f85950_18e2_411e_96f4_df42555386b8.slice/crio-9bef40075f364fede87963c1945c8c45a3fd749e4e3cd8a5011c1bb8c4327093 WatchSource:0}: Error finding container 9bef40075f364fede87963c1945c8c45a3fd749e4e3cd8a5011c1bb8c4327093: Status 404 returned error can't find the container with id 9bef40075f364fede87963c1945c8c45a3fd749e4e3cd8a5011c1bb8c4327093
Oct 30 03:06:30.224376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.224339    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"]
Oct 30 03:06:30.224828 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:30.224805    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba0cb4b_fe72_4878_9813_8ab4a3edb8d5.slice/crio-6981d30e1d63120e2aae938b224f8487a1ebc215b63f1b2d75329a8b7b26df12 WatchSource:0}: Error finding container 6981d30e1d63120e2aae938b224f8487a1ebc215b63f1b2d75329a8b7b26df12: Status 404 returned error can't find the container with id 6981d30e1d63120e2aae938b224f8487a1ebc215b63f1b2d75329a8b7b26df12
Oct 30 03:06:30.587895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.587793    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq" event={"ID":"b70c2e90-1d7e-46eb-b922-f40c0056f43d","Type":"ContainerStarted","Data":"b1630ea507d7a8e88457a949985a73785d0762f7ffb0b62d8b32911dc7810fd6"}
Oct 30 03:06:30.590484 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.590367    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj" event={"ID":"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5","Type":"ContainerStarted","Data":"6981d30e1d63120e2aae938b224f8487a1ebc215b63f1b2d75329a8b7b26df12"}
Oct 30 03:06:30.593512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.593437    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2" event={"ID":"011367c3-c75c-4875-bbe4-46ac08eb8382","Type":"ContainerStarted","Data":"35474418dba074febbb854528d413fc8f981c3e548641c5221f7ef67df4129b5"}
Oct 30 03:06:30.596317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:30.596249    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp" event={"ID":"56f85950-18e2-411e-96f4-df42555386b8","Type":"ContainerStarted","Data":"9bef40075f364fede87963c1945c8c45a3fd749e4e3cd8a5011c1bb8c4327093"}
Oct 30 03:06:31.372339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.372303    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"]
Oct 30 03:06:31.377521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.377492    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.406275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.406244    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"]
Oct 30 03:06:31.411292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.411269    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"]
Oct 30 03:06:31.415689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.414959    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.436646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.436623    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:06:31.441327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.441272    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"]
Oct 30 03:06:31.451236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.449797    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.451236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.449879    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.451236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.449934    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlzt\" (UniqueName: \"kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.477573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.475764    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"]
Oct 30 03:06:31.480030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.480007    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-h889m\""
Oct 30 03:06:31.480155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.480104    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fqfn6\""
Oct 30 03:06:31.480310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.480281    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.500677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.500636    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-qj5c5\""
Oct 30 03:06:31.501208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.501188    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"]
Oct 30 03:06:31.550600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550563    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.550740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550610    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfw9\" (UniqueName: \"kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.550740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550677    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlzt\" (UniqueName: \"kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.550843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550737    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.550843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550772    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.550843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550802    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.550843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550834    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.550993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550862    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6m9c\" (UniqueName: \"kubernetes.io/projected/d7badd71-7851-4ddb-b54b-86d2707078be-kube-api-access-t6m9c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.550993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550900    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7badd71-7851-4ddb-b54b-86d2707078be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.550993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550942    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.550993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.550968    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7badd71-7851-4ddb-b54b-86d2707078be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.551125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.551015    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.551125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.551046    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7badd71-7851-4ddb-b54b-86d2707078be-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.551125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.551089    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.551125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.551114    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.551258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.551189    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"]
Oct 30 03:06:31.552939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.552886    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.553285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.553260    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.556286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.555443    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.576809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.576654    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"]
Oct 30 03:06:31.586413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.586314    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlzt\" (UniqueName: \"kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.600265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.600184    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6mmp8\""
Oct 30 03:06:31.652482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652390    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.652482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652462    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652492    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6m9c\" (UniqueName: \"kubernetes.io/projected/d7badd71-7851-4ddb-b54b-86d2707078be-kube-api-access-t6m9c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652523    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7badd71-7851-4ddb-b54b-86d2707078be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652549    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652575    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652596    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7badd71-7851-4ddb-b54b-86d2707078be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652618    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652667    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652705    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.652753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652736    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7badd71-7851-4ddb-b54b-86d2707078be-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652764    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652788    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652817    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652843    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfb2\" (UniqueName: \"kubernetes.io/projected/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-kube-api-access-zjfb2\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652875    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652900    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652945    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.652975    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfw9\" (UniqueName: \"kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.653041    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.653323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.653077    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.653808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.653523    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.653808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.653765    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.656551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.654186    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.656551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.654489    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.656551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.654978    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.656551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.655217    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.656551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.656331    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7badd71-7851-4ddb-b54b-86d2707078be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.659927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.659858    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7badd71-7851-4ddb-b54b-86d2707078be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.662029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.662004    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7badd71-7851-4ddb-b54b-86d2707078be-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.687337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.687277    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfw9\" (UniqueName: \"kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.688028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.687979    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6m9c\" (UniqueName: \"kubernetes.io/projected/d7badd71-7851-4ddb-b54b-86d2707078be-kube-api-access-t6m9c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.689049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.689027    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7badd71-7851-4ddb-b54b-86d2707078be-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fl76j\" (UID: \"d7badd71-7851-4ddb-b54b-86d2707078be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.699374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.699355    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:06:31.740482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.740142    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754356    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754395    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754412    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754441    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754462    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfb2\" (UniqueName: \"kubernetes.io/projected/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-kube-api-access-zjfb2\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754496    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754580    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754653    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.754700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.754681    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.756116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.755831    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.756924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.756888    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.757418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.757394    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.757591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.757572    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.758204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.758161    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.761332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.761306    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.765896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.765874    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.786923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.785915    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.790695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.790628    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfb2\" (UniqueName: \"kubernetes.io/projected/cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c-kube-api-access-zjfb2\") pod \"openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm\" (UID: \"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:31.801305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.801274    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:06:31.876807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:31.876346    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:32.032698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.032659    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"]
Oct 30 03:06:32.070237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.070207    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"]
Oct 30 03:06:32.074481 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:32.074453    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7badd71_7851_4ddb_b54b_86d2707078be.slice/crio-ccc5767c605e633c4b47a5d2c8154bf3b962d29927d0660ec0cab0788d219307 WatchSource:0}: Error finding container ccc5767c605e633c4b47a5d2c8154bf3b962d29927d0660ec0cab0788d219307: Status 404 returned error can't find the container with id ccc5767c605e633c4b47a5d2c8154bf3b962d29927d0660ec0cab0788d219307
Oct 30 03:06:32.133128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.133087    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"]
Oct 30 03:06:32.134802 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:32.134764    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf49b073_f2c2_43f6_874d_0061aeeca38c.slice/crio-2e8498ff3d5de531df94fb52945d90eb2a465d4c61f907f7713f12df01f3e52e WatchSource:0}: Error finding container 2e8498ff3d5de531df94fb52945d90eb2a465d4c61f907f7713f12df01f3e52e: Status 404 returned error can't find the container with id 2e8498ff3d5de531df94fb52945d90eb2a465d4c61f907f7713f12df01f3e52e
Oct 30 03:06:32.223276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.223250    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"]
Oct 30 03:06:32.227205 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:06:32.227172    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2f1a6e_d6f6_4a88_bdbb_5104e0317d7c.slice/crio-33733878cb1d3c5e728aa78dd51ebf295edd6a4e15e359fa489ec2d4e4607669 WatchSource:0}: Error finding container 33733878cb1d3c5e728aa78dd51ebf295edd6a4e15e359fa489ec2d4e4607669: Status 404 returned error can't find the container with id 33733878cb1d3c5e728aa78dd51ebf295edd6a4e15e359fa489ec2d4e4607669
Oct 30 03:06:32.624165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.624042    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws" event={"ID":"53889185-3c4c-4862-93e1-820ec75abc42","Type":"ContainerStarted","Data":"9309df2ab0c0af9c7c452dfdd5dbac46eb2be975e4edc432be5e1c4bf192786e"}
Oct 30 03:06:32.630077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.629992    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" event={"ID":"d7badd71-7851-4ddb-b54b-86d2707078be","Type":"ContainerStarted","Data":"ccc5767c605e633c4b47a5d2c8154bf3b962d29927d0660ec0cab0788d219307"}
Oct 30 03:06:32.631727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.631696    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm" event={"ID":"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c","Type":"ContainerStarted","Data":"33733878cb1d3c5e728aa78dd51ebf295edd6a4e15e359fa489ec2d4e4607669"}
Oct 30 03:06:32.633071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:32.633010    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr" event={"ID":"af49b073-f2c2-43f6-874d-0061aeeca38c","Type":"ContainerStarted","Data":"2e8498ff3d5de531df94fb52945d90eb2a465d4c61f907f7713f12df01f3e52e"}
Oct 30 03:06:39.856331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:39.855609    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:06:39.856331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:39.855701    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:06:39.856331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:39.855744    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:06:40.288724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.286903    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:06:40.288724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.286994    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:06:40.288724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.287042    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:06:40.727544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.727395    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm" event={"ID":"cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c","Type":"ContainerStarted","Data":"a46b8567009ed46abdd30b15e58abdd3a8e08a56d6128d0053aab2790c28fd2b"}
Oct 30 03:06:40.733308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.733269    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" event={"ID":"d7badd71-7851-4ddb-b54b-86d2707078be","Type":"ContainerStarted","Data":"e0ac071e41e3523071fcd5a7a82e80808ef19baac3785315bd08faf6a2ff91d0"}
Oct 30 03:06:40.741160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.741136    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:40.828550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.828490    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podStartSLOduration=1.618897006 podStartE2EDuration="9.828475985s" totalImagesPullingTime="8.209578979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:40.82759674 +0000 UTC m=+229.256486159" watchObservedRunningTime="2026-10-30 03:06:40.828475985 +0000 UTC m=+229.257365400"
Oct 30 03:06:40.829077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.829035    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm" podStartSLOduration=2.204644406 podStartE2EDuration="9.829027847s" totalImagesPullingTime="7.624383441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:40.784882798 +0000 UTC m=+229.213772217" watchObservedRunningTime="2026-10-30 03:06:40.829027847 +0000 UTC m=+229.257917260"
Oct 30 03:06:40.879404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.877666    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:40.884542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:40.884180    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:41.742977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:41.742750    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:41.746844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:41.743800    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm"
Oct 30 03:06:41.748706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:41.748684    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:42.748624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:42.748588    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:42.749933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:42.749906    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:06:43.755489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:43.755380    4877 generic.go:356] "Generic (PLEG): container finished" podID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerID="ac18bcd18e846c62b06403104b031816f4503892614169329780b419e5e1408a" exitCode=0
Oct 30 03:06:43.755906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:43.755517    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l" event={"ID":"70f6e3e3-717c-4bff-80e2-7039cad092ea","Type":"ContainerDied","Data":"ac18bcd18e846c62b06403104b031816f4503892614169329780b419e5e1408a"}
Oct 30 03:06:43.758855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:43.758830    4877 generic.go:356] "Generic (PLEG): container finished" podID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerID="333818fc6ac83796d169a368ec01f932c004594375818442d44a8db0b4aac871" exitCode=0
Oct 30 03:06:43.758959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:43.758919    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr" event={"ID":"af49b073-f2c2-43f6-874d-0061aeeca38c","Type":"ContainerDied","Data":"333818fc6ac83796d169a368ec01f932c004594375818442d44a8db0b4aac871"}
Oct 30 03:06:43.761519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:43.761494    4877 generic.go:356] "Generic (PLEG): container finished" podID="53889185-3c4c-4862-93e1-820ec75abc42" containerID="df43df4c3a9884e2c500f8bed3ddd529839e124e4c1ebad1b9823dd6de0e2746" exitCode=0
Oct 30 03:06:43.763150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:43.763128    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws" event={"ID":"53889185-3c4c-4862-93e1-820ec75abc42","Type":"ContainerDied","Data":"df43df4c3a9884e2c500f8bed3ddd529839e124e4c1ebad1b9823dd6de0e2746"}
Oct 30 03:06:44.767513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:44.767410    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2" event={"ID":"011367c3-c75c-4875-bbe4-46ac08eb8382","Type":"ContainerStarted","Data":"cd56259c8ce4131177330cdd1a2e6bdc69425b564da65ff1126eda89cc2202ca"}
Oct 30 03:06:45.774620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.774581    4877 generic.go:356] "Generic (PLEG): container finished" podID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerID="1190a4816ca0257c2e34737ff824086f204d1224636a4118232d73a443a37262" exitCode=0
Oct 30 03:06:45.775084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.774639    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq" event={"ID":"b70c2e90-1d7e-46eb-b922-f40c0056f43d","Type":"ContainerDied","Data":"1190a4816ca0257c2e34737ff824086f204d1224636a4118232d73a443a37262"}
Oct 30 03:06:45.777305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.777275    4877 generic.go:356] "Generic (PLEG): container finished" podID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerID="707d2a17522dbbcea359018760ceb0b099a44037a91e6f3fe75af07f17e99017" exitCode=0
Oct 30 03:06:45.777455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.777356    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj" event={"ID":"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5","Type":"ContainerDied","Data":"707d2a17522dbbcea359018760ceb0b099a44037a91e6f3fe75af07f17e99017"}
Oct 30 03:06:45.780099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.780007    4877 generic.go:356] "Generic (PLEG): container finished" podID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerID="cd56259c8ce4131177330cdd1a2e6bdc69425b564da65ff1126eda89cc2202ca" exitCode=0
Oct 30 03:06:45.780174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.780122    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2" event={"ID":"011367c3-c75c-4875-bbe4-46ac08eb8382","Type":"ContainerDied","Data":"cd56259c8ce4131177330cdd1a2e6bdc69425b564da65ff1126eda89cc2202ca"}
Oct 30 03:06:45.784030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.783160    4877 generic.go:356] "Generic (PLEG): container finished" podID="53889185-3c4c-4862-93e1-820ec75abc42" containerID="2e10bc764a17f8128cd52598c407d799dfb06e0c70db5b4ec870688fac68bcfe" exitCode=0
Oct 30 03:06:45.784030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.783244    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws" event={"ID":"53889185-3c4c-4862-93e1-820ec75abc42","Type":"ContainerDied","Data":"2e10bc764a17f8128cd52598c407d799dfb06e0c70db5b4ec870688fac68bcfe"}
Oct 30 03:06:45.786664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.786639    4877 generic.go:356] "Generic (PLEG): container finished" podID="56f85950-18e2-411e-96f4-df42555386b8" containerID="a0302261560686c2866644d6dbf23d96aa8560fa499da9a6fa2458593de6e9cf" exitCode=0
Oct 30 03:06:45.786776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:45.786694    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp" event={"ID":"56f85950-18e2-411e-96f4-df42555386b8","Type":"ContainerDied","Data":"a0302261560686c2866644d6dbf23d96aa8560fa499da9a6fa2458593de6e9cf"}
Oct 30 03:06:46.238773 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:06:46.238681    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:06:47.805881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.805842    4877 generic.go:356] "Generic (PLEG): container finished" podID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerID="6cabd9d564f94b9c78d366da6739def667456cc6ae8c5272fa27bf1f03632da6" exitCode=0
Oct 30 03:06:47.806345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.805975    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2" event={"ID":"011367c3-c75c-4875-bbe4-46ac08eb8382","Type":"ContainerDied","Data":"6cabd9d564f94b9c78d366da6739def667456cc6ae8c5272fa27bf1f03632da6"}
Oct 30 03:06:47.808142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.808122    4877 generic.go:356] "Generic (PLEG): container finished" podID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerID="5f7931da8096f6236aebf7079353694727eb188144bf3e75dcedbc427f99e2f6" exitCode=0
Oct 30 03:06:47.808248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.808200    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l" event={"ID":"70f6e3e3-717c-4bff-80e2-7039cad092ea","Type":"ContainerDied","Data":"5f7931da8096f6236aebf7079353694727eb188144bf3e75dcedbc427f99e2f6"}
Oct 30 03:06:47.810224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.810154    4877 generic.go:356] "Generic (PLEG): container finished" podID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerID="1185c19eeaee2eff5623323f73df17182dde0d906e8ab7281bf5d9eae7c606d0" exitCode=0
Oct 30 03:06:47.810298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.810239    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr" event={"ID":"af49b073-f2c2-43f6-874d-0061aeeca38c","Type":"ContainerDied","Data":"1185c19eeaee2eff5623323f73df17182dde0d906e8ab7281bf5d9eae7c606d0"}
Oct 30 03:06:47.812670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.812650    4877 generic.go:356] "Generic (PLEG): container finished" podID="56f85950-18e2-411e-96f4-df42555386b8" containerID="049d5fe1a8f08819966e216c0783e62d04bb50f6a0c5328b739729ec9c4a97e0" exitCode=0
Oct 30 03:06:47.812752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.812725    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp" event={"ID":"56f85950-18e2-411e-96f4-df42555386b8","Type":"ContainerDied","Data":"049d5fe1a8f08819966e216c0783e62d04bb50f6a0c5328b739729ec9c4a97e0"}
Oct 30 03:06:47.815815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.815791    4877 generic.go:356] "Generic (PLEG): container finished" podID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerID="2600f121e8c66cdc61f81048379f9971db006dba5ed264f31211b0519547b9b7" exitCode=0
Oct 30 03:06:47.815920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.815814    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq" event={"ID":"b70c2e90-1d7e-46eb-b922-f40c0056f43d","Type":"ContainerDied","Data":"2600f121e8c66cdc61f81048379f9971db006dba5ed264f31211b0519547b9b7"}
Oct 30 03:06:47.818361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.818340    4877 generic.go:356] "Generic (PLEG): container finished" podID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerID="300f6d57a5e550717b1b48d7d887c868fa079beba91c6f501933fb6703f22002" exitCode=0
Oct 30 03:06:47.818456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:47.818371    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj" event={"ID":"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5","Type":"ContainerDied","Data":"300f6d57a5e550717b1b48d7d887c868fa079beba91c6f501933fb6703f22002"}
Oct 30 03:06:58.916134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:58.915579    4877 generic.go:356] "Generic (PLEG): container finished" podID="53889185-3c4c-4862-93e1-820ec75abc42" containerID="62ab99250fc885f51452e3e105194ead3c6d8a9aae7d7fbd89bab567b7d36406" exitCode=0
Oct 30 03:06:58.916134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:06:58.915642    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws" event={"ID":"53889185-3c4c-4862-93e1-820ec75abc42","Type":"ContainerDied","Data":"62ab99250fc885f51452e3e105194ead3c6d8a9aae7d7fbd89bab567b7d36406"}
Oct 30 03:07:00.131448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.131161    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:07:00.168068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.168032    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util\") pod \"53889185-3c4c-4862-93e1-820ec75abc42\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") "
Oct 30 03:07:00.168253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.168086    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt\" (UniqueName: \"kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt\") pod \"53889185-3c4c-4862-93e1-820ec75abc42\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") "
Oct 30 03:07:00.168253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.168153    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle\") pod \"53889185-3c4c-4862-93e1-820ec75abc42\" (UID: \"53889185-3c4c-4862-93e1-820ec75abc42\") "
Oct 30 03:07:00.170990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.170943    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt" pod "53889185-3c4c-4862-93e1-820ec75abc42" (UID: "53889185-3c4c-4862-93e1-820ec75abc42"). InnerVolumeSpecName "kube-api-access-hjlzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:00.171638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.171303    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle" pod "53889185-3c4c-4862-93e1-820ec75abc42" (UID: "53889185-3c4c-4862-93e1-820ec75abc42"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:00.181381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.181261    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util" pod "53889185-3c4c-4862-93e1-820ec75abc42" (UID: "53889185-3c4c-4862-93e1-820ec75abc42"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:00.269219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.269180    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:00.269219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.269223    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlzt\" (UniqueName: \"kubernetes.io/projected/53889185-3c4c-4862-93e1-820ec75abc42-kube-api-access-hjlzt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:00.269493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.269240    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53889185-3c4c-4862-93e1-820ec75abc42-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:00.946478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.944198    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws" event={"ID":"53889185-3c4c-4862-93e1-820ec75abc42","Type":"ContainerDied","Data":"9309df2ab0c0af9c7c452dfdd5dbac46eb2be975e4edc432be5e1c4bf192786e"}
Oct 30 03:07:00.946478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.944261    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9309df2ab0c0af9c7c452dfdd5dbac46eb2be975e4edc432be5e1c4bf192786e"
Oct 30 03:07:00.946478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:00.944354    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qtbws"
Oct 30 03:07:02.956619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.956583    4877 generic.go:356] "Generic (PLEG): container finished" podID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerID="0cf0b455472f6bd56c060a56a51ce7c67d526da7f071c55989839f81bbb86be3" exitCode=0
Oct 30 03:07:02.957065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.956667    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj" event={"ID":"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5","Type":"ContainerDied","Data":"0cf0b455472f6bd56c060a56a51ce7c67d526da7f071c55989839f81bbb86be3"}
Oct 30 03:07:02.958787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.958762    4877 generic.go:356] "Generic (PLEG): container finished" podID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerID="a452688afb52d4d590ba1365565356b9dbb2fd8f5b39c8d711029ce1c60fa5e5" exitCode=0
Oct 30 03:07:02.958903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.958866    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2" event={"ID":"011367c3-c75c-4875-bbe4-46ac08eb8382","Type":"ContainerDied","Data":"a452688afb52d4d590ba1365565356b9dbb2fd8f5b39c8d711029ce1c60fa5e5"}
Oct 30 03:07:02.960740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.960717    4877 generic.go:356] "Generic (PLEG): container finished" podID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerID="23642f157ada1339069bd9efb66efa8783eb5d243d86ed43af8104d3162b0cb9" exitCode=0
Oct 30 03:07:02.960853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.960780    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l" event={"ID":"70f6e3e3-717c-4bff-80e2-7039cad092ea","Type":"ContainerDied","Data":"23642f157ada1339069bd9efb66efa8783eb5d243d86ed43af8104d3162b0cb9"}
Oct 30 03:07:02.962802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.962774    4877 generic.go:356] "Generic (PLEG): container finished" podID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerID="abe6925ced45dd636a2045232697bd23a98b798fb1968440a76b48137a4724b8" exitCode=0
Oct 30 03:07:02.962890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.962866    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr" event={"ID":"af49b073-f2c2-43f6-874d-0061aeeca38c","Type":"ContainerDied","Data":"abe6925ced45dd636a2045232697bd23a98b798fb1968440a76b48137a4724b8"}
Oct 30 03:07:02.964729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.964708    4877 generic.go:356] "Generic (PLEG): container finished" podID="56f85950-18e2-411e-96f4-df42555386b8" containerID="20be9c78ff8e711214c6ea165598e37ebe50e273ff3757beb6157e872e585e48" exitCode=0
Oct 30 03:07:02.964835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.964793    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp" event={"ID":"56f85950-18e2-411e-96f4-df42555386b8","Type":"ContainerDied","Data":"20be9c78ff8e711214c6ea165598e37ebe50e273ff3757beb6157e872e585e48"}
Oct 30 03:07:02.966611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.966592    4877 generic.go:356] "Generic (PLEG): container finished" podID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerID="b3d6d2e967157e7bb35473c10848e43cbee183fea44c4eaed5e2298362a74a3d" exitCode=0
Oct 30 03:07:02.966674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:02.966625    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq" event={"ID":"b70c2e90-1d7e-46eb-b922-f40c0056f43d","Type":"ContainerDied","Data":"b3d6d2e967157e7bb35473c10848e43cbee183fea44c4eaed5e2298362a74a3d"}
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027210    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"]
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027658    4877 cpu_manager.go:414] "containerMap: removing container" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="extract"
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027673    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="extract"
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027697    4877 cpu_manager.go:414] "containerMap: removing container" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="pull"
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027701    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="pull"
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027721    4877 cpu_manager.go:414] "containerMap: removing container" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="util"
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027725    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="util"
Oct 30 03:07:04.028116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.027787    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="53889185-3c4c-4862-93e1-820ec75abc42" containerName="extract"
Oct 30 03:07:04.033448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.033402    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.049831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.049616    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 03:07:04.049831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.049799    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 03:07:04.061643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.061617    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 03:07:04.061794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.061664    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 03:07:04.061794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.061700    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2t4j2\""
Oct 30 03:07:04.061794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.061732    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 03:07:04.086046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.086008    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"]
Oct 30 03:07:04.104502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.104472    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsjb\" (UniqueName: \"kubernetes.io/projected/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-kube-api-access-rtsjb\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.104639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.104510    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.104639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.104603    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.104639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.104622    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.104805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.104646    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.104805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.104717    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.145680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.145633    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:07:04.206092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206063    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle\") pod \"56f85950-18e2-411e-96f4-df42555386b8\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") "
Oct 30 03:07:04.206206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206141    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45\" (UniqueName: \"kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45\") pod \"56f85950-18e2-411e-96f4-df42555386b8\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") "
Oct 30 03:07:04.206206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206175    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util\") pod \"56f85950-18e2-411e-96f4-df42555386b8\" (UID: \"56f85950-18e2-411e-96f4-df42555386b8\") "
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206331    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206415    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206465    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206506    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206619    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsjb\" (UniqueName: \"kubernetes.io/projected/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-kube-api-access-rtsjb\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.207013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.206905    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.208896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.208867    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle" pod "56f85950-18e2-411e-96f4-df42555386b8" (UID: "56f85950-18e2-411e-96f4-df42555386b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.211931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.211904    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.212226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.212203    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.213249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.212925    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.214772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.214743    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util" pod "56f85950-18e2-411e-96f4-df42555386b8" (UID: "56f85950-18e2-411e-96f4-df42555386b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.217016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.216989    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45" pod "56f85950-18e2-411e-96f4-df42555386b8" (UID: "56f85950-18e2-411e-96f4-df42555386b8"). InnerVolumeSpecName "kube-api-access-9rm45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:04.218098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.218077    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:07:04.219015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.218995    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.220354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.220332    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:07:04.224019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.223968    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:07:04.226606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.226587    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsjb\" (UniqueName: \"kubernetes.io/projected/086c4727-7f8a-4a23-9eeb-9f943a0d2f47-kube-api-access-rtsjb\") pod \"opendatahub-operator-controller-manager-5c85558465-q84h4\" (UID: \"086c4727-7f8a-4a23-9eeb-9f943a0d2f47\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.227538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.227521    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:07:04.230743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.230726    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:07:04.307246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307208    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle\") pod \"011367c3-c75c-4875-bbe4-46ac08eb8382\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") "
Oct 30 03:07:04.307455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307259    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util\") pod \"70f6e3e3-717c-4bff-80e2-7039cad092ea\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") "
Oct 30 03:07:04.307455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307290    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l\" (UniqueName: \"kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l\") pod \"011367c3-c75c-4875-bbe4-46ac08eb8382\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") "
Oct 30 03:07:04.307455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307317    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util\") pod \"af49b073-f2c2-43f6-874d-0061aeeca38c\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") "
Oct 30 03:07:04.307455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307361    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle\") pod \"70f6e3e3-717c-4bff-80e2-7039cad092ea\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") "
Oct 30 03:07:04.307455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307417    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle\") pod \"af49b073-f2c2-43f6-874d-0061aeeca38c\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") "
Oct 30 03:07:04.307648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307460    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util\") pod \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") "
Oct 30 03:07:04.307648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307487    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9\" (UniqueName: \"kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9\") pod \"af49b073-f2c2-43f6-874d-0061aeeca38c\" (UID: \"af49b073-f2c2-43f6-874d-0061aeeca38c\") "
Oct 30 03:07:04.307648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307518    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn\" (UniqueName: \"kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn\") pod \"70f6e3e3-717c-4bff-80e2-7039cad092ea\" (UID: \"70f6e3e3-717c-4bff-80e2-7039cad092ea\") "
Oct 30 03:07:04.307648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307586    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle\") pod \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") "
Oct 30 03:07:04.307648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307611    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util\") pod \"011367c3-c75c-4875-bbe4-46ac08eb8382\" (UID: \"011367c3-c75c-4875-bbe4-46ac08eb8382\") "
Oct 30 03:07:04.307648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307642    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util\") pod \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") "
Oct 30 03:07:04.307849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307675    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle\") pod \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") "
Oct 30 03:07:04.307849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307700    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth\" (UniqueName: \"kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth\") pod \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\" (UID: \"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5\") "
Oct 30 03:07:04.307849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307740    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf\" (UniqueName: \"kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf\") pod \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\" (UID: \"b70c2e90-1d7e-46eb-b922-f40c0056f43d\") "
Oct 30 03:07:04.308015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307941    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.308015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307953    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rm45\" (UniqueName: \"kubernetes.io/projected/56f85950-18e2-411e-96f4-df42555386b8-kube-api-access-9rm45\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.308015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.307963    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56f85950-18e2-411e-96f4-df42555386b8-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.308167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.308008    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle" pod "011367c3-c75c-4875-bbe4-46ac08eb8382" (UID: "011367c3-c75c-4875-bbe4-46ac08eb8382"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.309345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.309179    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle" pod "af49b073-f2c2-43f6-874d-0061aeeca38c" (UID: "af49b073-f2c2-43f6-874d-0061aeeca38c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.309345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.309300    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle" pod "8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" (UID: "8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.309656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.309615    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle" pod "70f6e3e3-717c-4bff-80e2-7039cad092ea" (UID: "70f6e3e3-717c-4bff-80e2-7039cad092ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.310244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.310221    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle" pod "b70c2e90-1d7e-46eb-b922-f40c0056f43d" (UID: "b70c2e90-1d7e-46eb-b922-f40c0056f43d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.314601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.314578    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf" pod "b70c2e90-1d7e-46eb-b922-f40c0056f43d" (UID: "b70c2e90-1d7e-46eb-b922-f40c0056f43d"). InnerVolumeSpecName "kube-api-access-g4vrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:04.315146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.315119    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util" pod "70f6e3e3-717c-4bff-80e2-7039cad092ea" (UID: "70f6e3e3-717c-4bff-80e2-7039cad092ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.316541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.316517    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util" pod "011367c3-c75c-4875-bbe4-46ac08eb8382" (UID: "011367c3-c75c-4875-bbe4-46ac08eb8382"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.316916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.316891    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn" pod "70f6e3e3-717c-4bff-80e2-7039cad092ea" (UID: "70f6e3e3-717c-4bff-80e2-7039cad092ea"). InnerVolumeSpecName "kube-api-access-l6zdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:04.316916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.316892    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util" pod "8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" (UID: "8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.317745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.317720    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9" pod "af49b073-f2c2-43f6-874d-0061aeeca38c" (UID: "af49b073-f2c2-43f6-874d-0061aeeca38c"). InnerVolumeSpecName "kube-api-access-lcfw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:04.317903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.317883    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util" pod "af49b073-f2c2-43f6-874d-0061aeeca38c" (UID: "af49b073-f2c2-43f6-874d-0061aeeca38c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.318829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.318809    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l" pod "011367c3-c75c-4875-bbe4-46ac08eb8382" (UID: "011367c3-c75c-4875-bbe4-46ac08eb8382"). InnerVolumeSpecName "kube-api-access-pkq2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:04.319235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.319213    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth" pod "8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" (UID: "8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5"). InnerVolumeSpecName "kube-api-access-2ttth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:04.319636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.319611    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util" pod "b70c2e90-1d7e-46eb-b922-f40c0056f43d" (UID: "b70c2e90-1d7e-46eb-b922-f40c0056f43d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:07:04.353751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.353727    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409557    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vrf\" (UniqueName: \"kubernetes.io/projected/b70c2e90-1d7e-46eb-b922-f40c0056f43d-kube-api-access-g4vrf\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409588    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409603    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409616    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkq2l\" (UniqueName: \"kubernetes.io/projected/011367c3-c75c-4875-bbe4-46ac08eb8382-kube-api-access-pkq2l\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409630    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409642    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f6e3e3-717c-4bff-80e2-7039cad092ea-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409654    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af49b073-f2c2-43f6-874d-0061aeeca38c-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409665    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409679    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcfw9\" (UniqueName: \"kubernetes.io/projected/af49b073-f2c2-43f6-874d-0061aeeca38c-kube-api-access-lcfw9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409691    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zdn\" (UniqueName: \"kubernetes.io/projected/70f6e3e3-717c-4bff-80e2-7039cad092ea-kube-api-access-l6zdn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409704    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b70c2e90-1d7e-46eb-b922-f40c0056f43d-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409717    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/011367c3-c75c-4875-bbe4-46ac08eb8382-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409728    4877 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-util\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409739    4877 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-bundle\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.412589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.409751    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttth\" (UniqueName: \"kubernetes.io/projected/8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5-kube-api-access-2ttth\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:04.713361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.713336    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"]
Oct 30 03:07:04.714770 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:04.714746    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086c4727_7f8a_4a23_9eeb_9f943a0d2f47.slice/crio-672ced902b41efa84f1096cad20b1d50e6c1ba367698368bf2dcfbb84896c6be WatchSource:0}: Error finding container 672ced902b41efa84f1096cad20b1d50e6c1ba367698368bf2dcfbb84896c6be: Status 404 returned error can't find the container with id 672ced902b41efa84f1096cad20b1d50e6c1ba367698368bf2dcfbb84896c6be
Oct 30 03:07:04.976949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.976873    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l" event={"ID":"70f6e3e3-717c-4bff-80e2-7039cad092ea","Type":"ContainerDied","Data":"85c37842980cfcd83cc21c5b03ecfeaa03533b347c92120cba03574bbebe668d"}
Oct 30 03:07:04.976949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.976923    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c37842980cfcd83cc21c5b03ecfeaa03533b347c92120cba03574bbebe668d"
Oct 30 03:07:04.976949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.976885    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs847l"
Oct 30 03:07:04.978795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.978768    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr" event={"ID":"af49b073-f2c2-43f6-874d-0061aeeca38c","Type":"ContainerDied","Data":"2e8498ff3d5de531df94fb52945d90eb2a465d4c61f907f7713f12df01f3e52e"}
Oct 30 03:07:04.978916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.978817    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8498ff3d5de531df94fb52945d90eb2a465d4c61f907f7713f12df01f3e52e"
Oct 30 03:07:04.978916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.978775    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vspgr"
Oct 30 03:07:04.980747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.980730    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp"
Oct 30 03:07:04.980834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.980754    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7szkdp" event={"ID":"56f85950-18e2-411e-96f4-df42555386b8","Type":"ContainerDied","Data":"9bef40075f364fede87963c1945c8c45a3fd749e4e3cd8a5011c1bb8c4327093"}
Oct 30 03:07:04.980834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.980795    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bef40075f364fede87963c1945c8c45a3fd749e4e3cd8a5011c1bb8c4327093"
Oct 30 03:07:04.982765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.982746    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq"
Oct 30 03:07:04.982863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.982778    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czmpcq" event={"ID":"b70c2e90-1d7e-46eb-b922-f40c0056f43d","Type":"ContainerDied","Data":"b1630ea507d7a8e88457a949985a73785d0762f7ffb0b62d8b32911dc7810fd6"}
Oct 30 03:07:04.982863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.982818    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1630ea507d7a8e88457a949985a73785d0762f7ffb0b62d8b32911dc7810fd6"
Oct 30 03:07:04.984918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.984900    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj" event={"ID":"8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5","Type":"ContainerDied","Data":"6981d30e1d63120e2aae938b224f8487a1ebc215b63f1b2d75329a8b7b26df12"}
Oct 30 03:07:04.984918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.984914    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q87lj"
Oct 30 03:07:04.985037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.984934    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6981d30e1d63120e2aae938b224f8487a1ebc215b63f1b2d75329a8b7b26df12"
Oct 30 03:07:04.987277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.987213    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4" event={"ID":"086c4727-7f8a-4a23-9eeb-9f943a0d2f47","Type":"ContainerStarted","Data":"672ced902b41efa84f1096cad20b1d50e6c1ba367698368bf2dcfbb84896c6be"}
Oct 30 03:07:04.989747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.989724    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2" event={"ID":"011367c3-c75c-4875-bbe4-46ac08eb8382","Type":"ContainerDied","Data":"35474418dba074febbb854528d413fc8f981c3e548641c5221f7ef67df4129b5"}
Oct 30 03:07:04.989824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.989772    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35474418dba074febbb854528d413fc8f981c3e548641c5221f7ef67df4129b5"
Oct 30 03:07:04.989824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:04.989732    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4lzh2"
Oct 30 03:07:08.005991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:08.005915    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4" event={"ID":"086c4727-7f8a-4a23-9eeb-9f943a0d2f47","Type":"ContainerStarted","Data":"ff293d85135e9e339f227230acdee0f89b9c1a5f5b92b3d7930693a013d23673"}
Oct 30 03:07:09.011935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:09.011899    4877 generic.go:356] "Generic (PLEG): container finished" podID="086c4727-7f8a-4a23-9eeb-9f943a0d2f47" containerID="ff293d85135e9e339f227230acdee0f89b9c1a5f5b92b3d7930693a013d23673" exitCode=0
Oct 30 03:07:09.012352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:09.011986    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4" event={"ID":"086c4727-7f8a-4a23-9eeb-9f943a0d2f47","Type":"ContainerDied","Data":"ff293d85135e9e339f227230acdee0f89b9c1a5f5b92b3d7930693a013d23673"}
Oct 30 03:07:10.023122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:10.023088    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4" event={"ID":"086c4727-7f8a-4a23-9eeb-9f943a0d2f47","Type":"ContainerStarted","Data":"2bd1752b2d7810a7c33e4475ad284c6c28f3e19146941050429111666b5df043"}
Oct 30 03:07:10.023556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:10.023210    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:10.103547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:10.103502    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4" podStartSLOduration=3.489817028 podStartE2EDuration="7.103491164s" totalImagesPullingTime="3.184003212s" totalInitContainerRuntime="429.670924ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:10.090979008 +0000 UTC m=+258.519868445" watchObservedRunningTime="2026-10-30 03:07:10.103491164 +0000 UTC m=+258.532380583"
Oct 30 03:07:16.227495 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:07:16.227458    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:07:19.659772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.659739    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"]
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660037    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="util"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660047    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="util"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660059    4877 cpu_manager.go:414] "containerMap: removing container" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660064    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660074    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660079    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660088    4877 cpu_manager.go:414] "containerMap: removing container" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="extract"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660092    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="extract"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660103    4877 cpu_manager.go:414] "containerMap: removing container" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="util"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660107    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="util"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660113    4877 cpu_manager.go:414] "containerMap: removing container" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660118    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660124    4877 cpu_manager.go:414] "containerMap: removing container" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="extract"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660128    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="extract"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660134    4877 cpu_manager.go:414] "containerMap: removing container" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660140    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660151    4877 cpu_manager.go:414] "containerMap: removing container" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="util"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660155    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="util"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660160    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660164    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660170    4877 cpu_manager.go:414] "containerMap: removing container" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660174    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="pull"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660181    4877 cpu_manager.go:414] "containerMap: removing container" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="extract"
Oct 30 03:07:19.660173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660185    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660194    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660198    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660204    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="util"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660209    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="util"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660214    4877 cpu_manager.go:414] "containerMap: removing container" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="util"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660218    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="util"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660224    4877 cpu_manager.go:414] "containerMap: removing container" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660227    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660234    4877 cpu_manager.go:414] "containerMap: removing container" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="util"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660238    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="util"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660243    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660246    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660291    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="af49b073-f2c2-43f6-874d-0061aeeca38c" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660299    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f6e3e3-717c-4bff-80e2-7039cad092ea" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660305    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba0cb4b-fe72-4878-9813-8ab4a3edb8d5" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660312    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f85950-18e2-411e-96f4-df42555386b8" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660317    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70c2e90-1d7e-46eb-b922-f40c0056f43d" containerName="extract"
Oct 30 03:07:19.660866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.660324    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="011367c3-c75c-4875-bbe4-46ac08eb8382" containerName="extract"
Oct 30 03:07:19.663494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.663474    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:19.691531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.691505    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 03:07:19.691531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.691527    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 03:07:19.691745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.691506    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 03:07:19.691745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.691644    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7kgmc\""
Oct 30 03:07:19.696553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.696531    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"]
Oct 30 03:07:19.729772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.729742    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv84x\" (UniqueName: \"kubernetes.io/projected/713ca209-955a-44dd-b634-18e58ef6735b-kube-api-access-wv84x\") pod \"dns-operator-controller-manager-8557cbdb84-gggrh\" (UID: \"713ca209-955a-44dd-b634-18e58ef6735b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:19.830704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.830670    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv84x\" (UniqueName: \"kubernetes.io/projected/713ca209-955a-44dd-b634-18e58ef6735b-kube-api-access-wv84x\") pod \"dns-operator-controller-manager-8557cbdb84-gggrh\" (UID: \"713ca209-955a-44dd-b634-18e58ef6735b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:19.850711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.850686    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv84x\" (UniqueName: \"kubernetes.io/projected/713ca209-955a-44dd-b634-18e58ef6735b-kube-api-access-wv84x\") pod \"dns-operator-controller-manager-8557cbdb84-gggrh\" (UID: \"713ca209-955a-44dd-b634-18e58ef6735b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:19.976261 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:19.976173    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:20.273654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:20.273626    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"]
Oct 30 03:07:20.274094 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:20.274074    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713ca209_955a_44dd_b634_18e58ef6735b.slice/crio-cd1a45bdc38e4bc80928e54d4097072c61095faed2613cf1b6c911776556418b WatchSource:0}: Error finding container cd1a45bdc38e4bc80928e54d4097072c61095faed2613cf1b6c911776556418b: Status 404 returned error can't find the container with id cd1a45bdc38e4bc80928e54d4097072c61095faed2613cf1b6c911776556418b
Oct 30 03:07:21.030462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:21.030411    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-q84h4"
Oct 30 03:07:21.073518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:21.073485    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh" event={"ID":"713ca209-955a-44dd-b634-18e58ef6735b","Type":"ContainerStarted","Data":"cd1a45bdc38e4bc80928e54d4097072c61095faed2613cf1b6c911776556418b"}
Oct 30 03:07:23.963099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:23.963062    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"]
Oct 30 03:07:23.966558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:23.966535    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:23.999857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:23.999836    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2lfn8\""
Oct 30 03:07:24.001537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.001518    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"]
Oct 30 03:07:24.067298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.067272    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctfc\" (UniqueName: \"kubernetes.io/projected/66695834-c9cd-430e-946c-aa57ae794bca-kube-api-access-kctfc\") pod \"limitador-operator-controller-manager-5778bb4449-rcpz6\" (UID: \"66695834-c9cd-430e-946c-aa57ae794bca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:24.096520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.096489    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh" event={"ID":"713ca209-955a-44dd-b634-18e58ef6735b","Type":"ContainerStarted","Data":"d7665a6f85fd6f61c8507a33317e35a75d3582363c188902a5b6ee8835225b97"}
Oct 30 03:07:24.096643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.096623    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:24.142236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.142191    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh" podStartSLOduration=2.387934314 podStartE2EDuration="5.142178592s" totalImagesPullingTime="2.754244278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:24.140931327 +0000 UTC m=+272.569820759" watchObservedRunningTime="2026-10-30 03:07:24.142178592 +0000 UTC m=+272.571068010"
Oct 30 03:07:24.167874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.167847    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctfc\" (UniqueName: \"kubernetes.io/projected/66695834-c9cd-430e-946c-aa57ae794bca-kube-api-access-kctfc\") pod \"limitador-operator-controller-manager-5778bb4449-rcpz6\" (UID: \"66695834-c9cd-430e-946c-aa57ae794bca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:24.189624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.189599    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctfc\" (UniqueName: \"kubernetes.io/projected/66695834-c9cd-430e-946c-aa57ae794bca-kube-api-access-kctfc\") pod \"limitador-operator-controller-manager-5778bb4449-rcpz6\" (UID: \"66695834-c9cd-430e-946c-aa57ae794bca\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:24.278704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.278679    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:24.580244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:24.580217    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"]
Oct 30 03:07:24.580459 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:24.580431    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66695834_c9cd_430e_946c_aa57ae794bca.slice/crio-d5a7a3fd8c9b84fb4fdd99b17816ca775f545d3ab360b26efc6fa8865105651a WatchSource:0}: Error finding container d5a7a3fd8c9b84fb4fdd99b17816ca775f545d3ab360b26efc6fa8865105651a: Status 404 returned error can't find the container with id d5a7a3fd8c9b84fb4fdd99b17816ca775f545d3ab360b26efc6fa8865105651a
Oct 30 03:07:25.102509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.102463    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6" event={"ID":"66695834-c9cd-430e-946c-aa57ae794bca","Type":"ContainerStarted","Data":"d5a7a3fd8c9b84fb4fdd99b17816ca775f545d3ab360b26efc6fa8865105651a"}
Oct 30 03:07:25.903978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.903940    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"]
Oct 30 03:07:25.907935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.907909    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:25.922835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.922806    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"]
Oct 30 03:07:25.933357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.933334    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:07:25.933542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.933523    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 03:07:25.933644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.933625    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-g8x4j\""
Oct 30 03:07:25.984906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.984880    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhcq\" (UniqueName: \"kubernetes.io/projected/8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b-kube-api-access-zbhcq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj\" (UID: \"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:25.985020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:25.984920    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj\" (UID: \"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:26.086065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.086035    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhcq\" (UniqueName: \"kubernetes.io/projected/8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b-kube-api-access-zbhcq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj\" (UID: \"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:26.086226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.086076    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj\" (UID: \"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:26.086458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.086442    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj\" (UID: \"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:26.107028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.107003    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhcq\" (UniqueName: \"kubernetes.io/projected/8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b-kube-api-access-zbhcq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj\" (UID: \"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:26.108703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.108675    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6" event={"ID":"66695834-c9cd-430e-946c-aa57ae794bca","Type":"ContainerStarted","Data":"7110997673ce9dc42b118092cbe79691135a5f299dbee2acd967e02b7c79d08c"}
Oct 30 03:07:26.108824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.108806    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:26.157475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.157366    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6" podStartSLOduration=1.7558512149999999 podStartE2EDuration="3.15735321s" totalImagesPullingTime="1.401501995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:26.155394262 +0000 UTC m=+274.584283700" watchObservedRunningTime="2026-10-30 03:07:26.15735321 +0000 UTC m=+274.586242628"
Oct 30 03:07:26.221363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.221330    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"
Oct 30 03:07:26.551362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:26.551333    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj"]
Oct 30 03:07:26.552142 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:26.552112    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf0df5e_ebc2_4c91_a5e5_752e6b138e3b.slice/crio-4634eb2eaebca5a62fa5c1f0fded24db35d22495a9c46a847af5db9284d97f1d WatchSource:0}: Error finding container 4634eb2eaebca5a62fa5c1f0fded24db35d22495a9c46a847af5db9284d97f1d: Status 404 returned error can't find the container with id 4634eb2eaebca5a62fa5c1f0fded24db35d22495a9c46a847af5db9284d97f1d
Oct 30 03:07:27.114453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:27.114400    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj" event={"ID":"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b","Type":"ContainerStarted","Data":"4634eb2eaebca5a62fa5c1f0fded24db35d22495a9c46a847af5db9284d97f1d"}
Oct 30 03:07:29.124476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:29.124442    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj" event={"ID":"8bf0df5e-ebc2-4c91-a5e5-752e6b138e3b","Type":"ContainerStarted","Data":"4cd4ccfcb041873fd5101bca25fe302caec124c09e541e3a62a8b1ffbd3dfe58"}
Oct 30 03:07:29.198131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:29.198079    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lbdjj" podStartSLOduration=2.683509676 podStartE2EDuration="4.198060464s" totalImagesPullingTime="1.514550788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:29.185485821 +0000 UTC m=+277.614375239" watchObservedRunningTime="2026-10-30 03:07:29.198060464 +0000 UTC m=+277.626949884"
Oct 30 03:07:32.171446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.171403    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"]
Oct 30 03:07:32.175244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.175221    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.206703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.206678    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"]
Oct 30 03:07:32.235985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.235956    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-txkl6\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.236142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.236025    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxsv\" (UniqueName: \"kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv\") pod \"kuadrant-operator-controller-manager-678d87766f-txkl6\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.258491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.258461    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cqs85\""
Oct 30 03:07:32.337383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.337347    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-txkl6\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.337596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.337434    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxsv\" (UniqueName: \"kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv\") pod \"kuadrant-operator-controller-manager-678d87766f-txkl6\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.337800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.337780    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-txkl6\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.366228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.366201    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxsv\" (UniqueName: \"kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv\") pod \"kuadrant-operator-controller-manager-678d87766f-txkl6\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.487369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.487273    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:32.836086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:32.836055    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"]
Oct 30 03:07:32.849126 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:32.849085    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadde55dc_547d_42d9_9cc6_7e281e545561.slice/crio-c3a5ad836c8d1446a7ac3778ca9d7c1d8f109696a45381e05d4a8951ffde5093 WatchSource:0}: Error finding container c3a5ad836c8d1446a7ac3778ca9d7c1d8f109696a45381e05d4a8951ffde5093: Status 404 returned error can't find the container with id c3a5ad836c8d1446a7ac3778ca9d7c1d8f109696a45381e05d4a8951ffde5093
Oct 30 03:07:33.142796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:33.142708    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6" event={"ID":"adde55dc-547d-42d9-9cc6-7e281e545561","Type":"ContainerStarted","Data":"c3a5ad836c8d1446a7ac3778ca9d7c1d8f109696a45381e05d4a8951ffde5093"}
Oct 30 03:07:35.105794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:35.105758    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gggrh"
Oct 30 03:07:36.060151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.060114    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"]
Oct 30 03:07:36.063897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.063873    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:36.088558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.088529    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"]
Oct 30 03:07:36.098646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.098623    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bqw4k\""
Oct 30 03:07:36.170989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.170948    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9ct\" (UniqueName: \"kubernetes.io/projected/297ec4ff-cbaf-48ea-a5cd-fab91d07a75e-kube-api-access-2b9ct\") pod \"authorino-operator-7b7bb4bc77-npmx5\" (UID: \"297ec4ff-cbaf-48ea-a5cd-fab91d07a75e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:36.272720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.272494    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9ct\" (UniqueName: \"kubernetes.io/projected/297ec4ff-cbaf-48ea-a5cd-fab91d07a75e-kube-api-access-2b9ct\") pod \"authorino-operator-7b7bb4bc77-npmx5\" (UID: \"297ec4ff-cbaf-48ea-a5cd-fab91d07a75e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:36.294367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.294334    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9ct\" (UniqueName: \"kubernetes.io/projected/297ec4ff-cbaf-48ea-a5cd-fab91d07a75e-kube-api-access-2b9ct\") pod \"authorino-operator-7b7bb4bc77-npmx5\" (UID: \"297ec4ff-cbaf-48ea-a5cd-fab91d07a75e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:36.378741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.378667    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:36.719682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:36.719654    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"]
Oct 30 03:07:36.720789 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:36.720763    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297ec4ff_cbaf_48ea_a5cd_fab91d07a75e.slice/crio-93663ad96e13e26b7e61bbebdbc5d06ac5d606c2ed32b27ec7edf701bb542731 WatchSource:0}: Error finding container 93663ad96e13e26b7e61bbebdbc5d06ac5d606c2ed32b27ec7edf701bb542731: Status 404 returned error can't find the container with id 93663ad96e13e26b7e61bbebdbc5d06ac5d606c2ed32b27ec7edf701bb542731
Oct 30 03:07:37.117173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.117139    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rcpz6"
Oct 30 03:07:37.165851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.165808    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6" event={"ID":"adde55dc-547d-42d9-9cc6-7e281e545561","Type":"ContainerStarted","Data":"977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a"}
Oct 30 03:07:37.166057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.165881    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:37.167562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.167535    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5" event={"ID":"297ec4ff-cbaf-48ea-a5cd-fab91d07a75e","Type":"ContainerStarted","Data":"93663ad96e13e26b7e61bbebdbc5d06ac5d606c2ed32b27ec7edf701bb542731"}
Oct 30 03:07:37.213569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.213519    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6" podStartSLOduration=1.627609455 podStartE2EDuration="5.213505483s" totalImagesPullingTime="3.585896028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:37.212235409 +0000 UTC m=+285.641124841" watchObservedRunningTime="2026-10-30 03:07:37.213505483 +0000 UTC m=+285.642394901"
Oct 30 03:07:37.800977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.800939    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"]
Oct 30 03:07:37.804623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.804599    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.833333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.833302    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"]
Oct 30 03:07:37.863259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.863227    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k8dw5\""
Oct 30 03:07:37.887797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.887762    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.887944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.887814    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8rk\" (UniqueName: \"kubernetes.io/projected/1a4cb6aa-0810-497b-a9c8-510595534f09-kube-api-access-rp8rk\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.887944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.887835    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.887944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.887920    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.888043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.887955    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.888043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.887989    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.888043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.888010    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.888043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.888029    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.888164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.888048    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb6aa-0810-497b-a9c8-510595534f09-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989332    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8rk\" (UniqueName: \"kubernetes.io/projected/1a4cb6aa-0810-497b-a9c8-510595534f09-kube-api-access-rp8rk\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989365    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989390    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989474    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989547    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989577    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989603    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989637    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb6aa-0810-497b-a9c8-510595534f09-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.989970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.989919    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.990216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.990112    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.990216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.990176    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.990500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.990478    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.990604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.990477    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.990662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.990634    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb6aa-0810-497b-a9c8-510595534f09-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.992322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.992296    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:37.992466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:37.992338    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:38.011865 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.011839    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb6aa-0810-497b-a9c8-510595534f09-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:38.013461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.012397    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8rk\" (UniqueName: \"kubernetes.io/projected/1a4cb6aa-0810-497b-a9c8-510595534f09-kube-api-access-rp8rk\") pod \"data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn\" (UID: \"1a4cb6aa-0810-497b-a9c8-510595534f09\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:38.122639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.122546    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:38.178931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.178892    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5" event={"ID":"297ec4ff-cbaf-48ea-a5cd-fab91d07a75e","Type":"ContainerStarted","Data":"d9c7a9d756574bde6ac1307f91d041a411f22727d0286ab3a07ee82f56084f29"}
Oct 30 03:07:38.179081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.179022    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:38.222631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.222563    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5" podStartSLOduration=0.990357175 podStartE2EDuration="2.22254288s" totalImagesPullingTime="1.232185705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:38.220614454 +0000 UTC m=+286.649503872" watchObservedRunningTime="2026-10-30 03:07:38.22254288 +0000 UTC m=+286.651432299"
Oct 30 03:07:38.452054 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:38.452024    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4cb6aa_0810_497b_a9c8_510595534f09.slice/crio-ac52e7bd65e2841232921e5a44394d21e8490644e34452d81df8a3a0d2d0c6db WatchSource:0}: Error finding container ac52e7bd65e2841232921e5a44394d21e8490644e34452d81df8a3a0d2d0c6db: Status 404 returned error can't find the container with id ac52e7bd65e2841232921e5a44394d21e8490644e34452d81df8a3a0d2d0c6db
Oct 30 03:07:38.452480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.452458    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"]
Oct 30 03:07:38.454513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.454487    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:07:38.454576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.454549    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:07:38.454614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:38.454585    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:07:39.184903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:39.184861    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn" event={"ID":"1a4cb6aa-0810-497b-a9c8-510595534f09","Type":"ContainerStarted","Data":"89095292d9db5b361f8fb86fdfd7f4849ae523c943486ff1458ea570be89e8a8"}
Oct 30 03:07:39.185104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:39.184930    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn" event={"ID":"1a4cb6aa-0810-497b-a9c8-510595534f09","Type":"ContainerStarted","Data":"ac52e7bd65e2841232921e5a44394d21e8490644e34452d81df8a3a0d2d0c6db"}
Oct 30 03:07:39.231749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:39.231693    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn" podStartSLOduration=2.231679919 podStartE2EDuration="2.231679919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:39.229604928 +0000 UTC m=+287.658494351" watchObservedRunningTime="2026-10-30 03:07:39.231679919 +0000 UTC m=+287.660569336"
Oct 30 03:07:40.035027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.034986    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bfg2n"]
Oct 30 03:07:40.038471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.038455    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:40.056887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.056859    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bfg2n"]
Oct 30 03:07:40.066904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.066881    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dd2cq\""
Oct 30 03:07:40.123109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.123086    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:40.127859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.127829    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:40.189595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.189558    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:40.190333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.190316    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn"
Oct 30 03:07:40.206603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.206580    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt429\" (UniqueName: \"kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429\") pod \"authorino-57f7f8659f-bfg2n\" (UID: \"3e03dded-6a60-422b-82b7-e99621af576c\") " pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:40.307953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.307524    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt429\" (UniqueName: \"kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429\") pod \"authorino-57f7f8659f-bfg2n\" (UID: \"3e03dded-6a60-422b-82b7-e99621af576c\") " pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:40.327796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.327762    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt429\" (UniqueName: \"kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429\") pod \"authorino-57f7f8659f-bfg2n\" (UID: \"3e03dded-6a60-422b-82b7-e99621af576c\") " pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:40.365406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.365370    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:40.666823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.666782    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bfg2n"]
Oct 30 03:07:40.666998 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:40.666972    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e03dded_6a60_422b_82b7_e99621af576c.slice/crio-6c301ca0bc2d52ca0c2e8ee2a36e40c9e1294ddde5dd3ada93ac0ba3aa071e26 WatchSource:0}: Error finding container 6c301ca0bc2d52ca0c2e8ee2a36e40c9e1294ddde5dd3ada93ac0ba3aa071e26: Status 404 returned error can't find the container with id 6c301ca0bc2d52ca0c2e8ee2a36e40c9e1294ddde5dd3ada93ac0ba3aa071e26
Oct 30 03:07:40.737510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.737474    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"]
Oct 30 03:07:40.786054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.786013    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"]
Oct 30 03:07:40.786229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.786128    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:40.800628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.800598    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 03:07:40.829164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.829137    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 03:07:40.829385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.829372    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-clchq\""
Oct 30 03:07:40.913378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.913348    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/490cd1fd-4982-4de2-824f-1a3f359dc24d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:40.913538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.913384    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/490cd1fd-4982-4de2-824f-1a3f359dc24d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:40.913538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:40.913491    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccfh\" (UniqueName: \"kubernetes.io/projected/490cd1fd-4982-4de2-824f-1a3f359dc24d-kube-api-access-qccfh\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.014479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.014375    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/490cd1fd-4982-4de2-824f-1a3f359dc24d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.014479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.014409    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/490cd1fd-4982-4de2-824f-1a3f359dc24d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.014479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.014467    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccfh\" (UniqueName: \"kubernetes.io/projected/490cd1fd-4982-4de2-824f-1a3f359dc24d-kube-api-access-qccfh\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.015180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.015160    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/490cd1fd-4982-4de2-824f-1a3f359dc24d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.016897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.016861    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/490cd1fd-4982-4de2-824f-1a3f359dc24d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.033859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.033833    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccfh\" (UniqueName: \"kubernetes.io/projected/490cd1fd-4982-4de2-824f-1a3f359dc24d-kube-api-access-qccfh\") pod \"kuadrant-console-plugin-66c4fb575d-8m78v\" (UID: \"490cd1fd-4982-4de2-824f-1a3f359dc24d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.115752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.115721    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"
Oct 30 03:07:41.198978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.198941    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bfg2n" event={"ID":"3e03dded-6a60-422b-82b7-e99621af576c","Type":"ContainerStarted","Data":"6c301ca0bc2d52ca0c2e8ee2a36e40c9e1294ddde5dd3ada93ac0ba3aa071e26"}
Oct 30 03:07:41.436705 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:41.436671    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod490cd1fd_4982_4de2_824f_1a3f359dc24d.slice/crio-3974a0c8155b13d5d86031fef6eea6b696256ae55312cdca4349c904e70a99bc WatchSource:0}: Error finding container 3974a0c8155b13d5d86031fef6eea6b696256ae55312cdca4349c904e70a99bc: Status 404 returned error can't find the container with id 3974a0c8155b13d5d86031fef6eea6b696256ae55312cdca4349c904e70a99bc
Oct 30 03:07:41.437047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.436998    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v"]
Oct 30 03:07:41.835736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.835703    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-4tw4m"]
Oct 30 03:07:41.838901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.838870    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:41.853613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.853584    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 03:07:41.853727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.853652    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 03:07:41.868395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.868370    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4tw4m"]
Oct 30 03:07:41.885936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.885909    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-g8h4z\""
Oct 30 03:07:41.921166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.921135    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7e36b6-64e3-471f-87b4-e394c1fc7d37-bound-sa-token\") pod \"cert-manager-597b95cb84-4tw4m\" (UID: \"ed7e36b6-64e3-471f-87b4-e394c1fc7d37\") " pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:41.921312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:41.921212    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g265\" (UniqueName: \"kubernetes.io/projected/ed7e36b6-64e3-471f-87b4-e394c1fc7d37-kube-api-access-2g265\") pod \"cert-manager-597b95cb84-4tw4m\" (UID: \"ed7e36b6-64e3-471f-87b4-e394c1fc7d37\") " pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:42.022668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.022623    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g265\" (UniqueName: \"kubernetes.io/projected/ed7e36b6-64e3-471f-87b4-e394c1fc7d37-kube-api-access-2g265\") pod \"cert-manager-597b95cb84-4tw4m\" (UID: \"ed7e36b6-64e3-471f-87b4-e394c1fc7d37\") " pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:42.022868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.022761    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7e36b6-64e3-471f-87b4-e394c1fc7d37-bound-sa-token\") pod \"cert-manager-597b95cb84-4tw4m\" (UID: \"ed7e36b6-64e3-471f-87b4-e394c1fc7d37\") " pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:42.045789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.045761    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g265\" (UniqueName: \"kubernetes.io/projected/ed7e36b6-64e3-471f-87b4-e394c1fc7d37-kube-api-access-2g265\") pod \"cert-manager-597b95cb84-4tw4m\" (UID: \"ed7e36b6-64e3-471f-87b4-e394c1fc7d37\") " pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:42.045939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.045790    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed7e36b6-64e3-471f-87b4-e394c1fc7d37-bound-sa-token\") pod \"cert-manager-597b95cb84-4tw4m\" (UID: \"ed7e36b6-64e3-471f-87b4-e394c1fc7d37\") " pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:42.152598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.152516    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4tw4m"
Oct 30 03:07:42.208330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.208294    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v" event={"ID":"490cd1fd-4982-4de2-824f-1a3f359dc24d","Type":"ContainerStarted","Data":"3974a0c8155b13d5d86031fef6eea6b696256ae55312cdca4349c904e70a99bc"}
Oct 30 03:07:42.471488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:42.465124    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4tw4m"]
Oct 30 03:07:42.471488 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:07:42.466368    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded7e36b6_64e3_471f_87b4_e394c1fc7d37.slice/crio-368a2bb830d7c1a92c26463ee9afed0c473efda1b007a33bc2ccbadb94f5aae1 WatchSource:0}: Error finding container 368a2bb830d7c1a92c26463ee9afed0c473efda1b007a33bc2ccbadb94f5aae1: Status 404 returned error can't find the container with id 368a2bb830d7c1a92c26463ee9afed0c473efda1b007a33bc2ccbadb94f5aae1
Oct 30 03:07:43.220180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:43.220142    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4tw4m" event={"ID":"ed7e36b6-64e3-471f-87b4-e394c1fc7d37","Type":"ContainerStarted","Data":"368a2bb830d7c1a92c26463ee9afed0c473efda1b007a33bc2ccbadb94f5aae1"}
Oct 30 03:07:43.676503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:43.676398    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bfg2n"]
Oct 30 03:07:44.228445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:44.228373    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bfg2n" event={"ID":"3e03dded-6a60-422b-82b7-e99621af576c","Type":"ContainerStarted","Data":"0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a"}
Oct 30 03:07:44.269585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:44.269524    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-bfg2n" podStartSLOduration=1.785640716 podStartE2EDuration="4.269506753s" totalImagesPullingTime="2.483866037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:44.267546217 +0000 UTC m=+292.696435662" watchObservedRunningTime="2026-10-30 03:07:44.269506753 +0000 UTC m=+292.698396170"
Oct 30 03:07:45.234204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:45.234125    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-bfg2n" podUID="3e03dded-6a60-422b-82b7-e99621af576c" containerName="authorino" containerID="cri-o://0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a" gracePeriod=30
Oct 30 03:07:45.496391 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:45.496335    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:45.657734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:45.657698    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429\" (UniqueName: \"kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429\") pod \"3e03dded-6a60-422b-82b7-e99621af576c\" (UID: \"3e03dded-6a60-422b-82b7-e99621af576c\") "
Oct 30 03:07:45.660043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:45.660016    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429" pod "3e03dded-6a60-422b-82b7-e99621af576c" (UID: "3e03dded-6a60-422b-82b7-e99621af576c"). InnerVolumeSpecName "kube-api-access-lt429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:07:45.759060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:45.758969    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt429\" (UniqueName: \"kubernetes.io/projected/3e03dded-6a60-422b-82b7-e99621af576c-kube-api-access-lt429\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:07:46.233438 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:07:46.233381    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:07:46.241969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.241929    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4tw4m" event={"ID":"ed7e36b6-64e3-471f-87b4-e394c1fc7d37","Type":"ContainerStarted","Data":"ec13d184e3ee94a1030f87355e6e90db567612a6dbeeff6f7c2628ac9748029a"}
Oct 30 03:07:46.243540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.243513    4877 generic.go:356] "Generic (PLEG): container finished" podID="3e03dded-6a60-422b-82b7-e99621af576c" containerID="0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a" exitCode=0
Oct 30 03:07:46.243662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.243644    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bfg2n" event={"ID":"3e03dded-6a60-422b-82b7-e99621af576c","Type":"ContainerDied","Data":"0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a"}
Oct 30 03:07:46.243735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.243681    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-bfg2n" event={"ID":"3e03dded-6a60-422b-82b7-e99621af576c","Type":"ContainerDied","Data":"6c301ca0bc2d52ca0c2e8ee2a36e40c9e1294ddde5dd3ada93ac0ba3aa071e26"}
Oct 30 03:07:46.243735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.243706    4877 scope.go:122] "RemoveContainer" containerID="0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a"
Oct 30 03:07:46.243965 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.243950    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-bfg2n"
Oct 30 03:07:46.253500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.253480    4877 scope.go:122] "RemoveContainer" containerID="0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a"
Oct 30 03:07:46.253822 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:07:46.253799    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a\": container with ID starting with 0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a not found: ID does not exist" containerID="0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a"
Oct 30 03:07:46.253896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.253832    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a"} err="failed to get container status \"0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a\": rpc error: code = NotFound desc = could not find container \"0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a\": container with ID starting with 0fd7b87d0b5b34f8d359a1730ebf2d15bdbd60d8a20c13fa2889ec67b5ec088a not found: ID does not exist"
Oct 30 03:07:46.282850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.282799    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-4tw4m" podStartSLOduration=2.478504604 podStartE2EDuration="5.282783639s" totalImagesPullingTime="2.804279035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:46.281661713 +0000 UTC m=+294.710551151" watchObservedRunningTime="2026-10-30 03:07:46.282783639 +0000 UTC m=+294.711673058"
Oct 30 03:07:46.319739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.319711    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bfg2n"]
Oct 30 03:07:46.322964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:46.322940    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-bfg2n"]
Oct 30 03:07:47.251941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:47.251898    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v" event={"ID":"490cd1fd-4982-4de2-824f-1a3f359dc24d","Type":"ContainerStarted","Data":"4f552fc85b1a152a5f76e9c4735506d87fff73329971bcb394565e6bde3cdbf6"}
Oct 30 03:07:47.295441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:47.295354    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8m78v" podStartSLOduration=2.262949539 podStartE2EDuration="7.295338108s" totalImagesPullingTime="5.032388569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:47.293844286 +0000 UTC m=+295.722733714" watchObservedRunningTime="2026-10-30 03:07:47.295338108 +0000 UTC m=+295.724227526"
Oct 30 03:07:47.696158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:47.696122    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e03dded-6a60-422b-82b7-e99621af576c" path="/var/lib/kubelet/pods/3e03dded-6a60-422b-82b7-e99621af576c/volumes"
Oct 30 03:07:48.180359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:48.180328    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 03:07:50.192498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:50.192464    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-npmx5"
Oct 30 03:07:51.728022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:51.727991    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:07:51.728525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:07:51.728448    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:08:00.045343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.045311    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"]
Oct 30 03:08:00.045908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.045794    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3e03dded-6a60-422b-82b7-e99621af576c" containerName="authorino"
Oct 30 03:08:00.045908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.045818    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e03dded-6a60-422b-82b7-e99621af576c" containerName="authorino"
Oct 30 03:08:00.045992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.045936    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e03dded-6a60-422b-82b7-e99621af576c" containerName="authorino"
Oct 30 03:08:00.051275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.051256    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.067617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.067599    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 03:08:00.079444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.079401    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gd7n2\""
Oct 30 03:08:00.079556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.079415    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 03:08:00.079617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.079601    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 03:08:00.081897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.081876    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnqr\" (UniqueName: \"kubernetes.io/projected/a51876bb-942d-483d-9cda-5372b1b0399f-kube-api-access-cwnqr\") pod \"postgres-7cdbf6dc44-9pfkh\" (UID: \"a51876bb-942d-483d-9cda-5372b1b0399f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.081997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.081931    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a51876bb-942d-483d-9cda-5372b1b0399f-data\") pod \"postgres-7cdbf6dc44-9pfkh\" (UID: \"a51876bb-942d-483d-9cda-5372b1b0399f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.082225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.082203    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"]
Oct 30 03:08:00.183556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.183517    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnqr\" (UniqueName: \"kubernetes.io/projected/a51876bb-942d-483d-9cda-5372b1b0399f-kube-api-access-cwnqr\") pod \"postgres-7cdbf6dc44-9pfkh\" (UID: \"a51876bb-942d-483d-9cda-5372b1b0399f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.183556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.183600    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a51876bb-942d-483d-9cda-5372b1b0399f-data\") pod \"postgres-7cdbf6dc44-9pfkh\" (UID: \"a51876bb-942d-483d-9cda-5372b1b0399f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.184076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.184058    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a51876bb-942d-483d-9cda-5372b1b0399f-data\") pod \"postgres-7cdbf6dc44-9pfkh\" (UID: \"a51876bb-942d-483d-9cda-5372b1b0399f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.202749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.202723    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnqr\" (UniqueName: \"kubernetes.io/projected/a51876bb-942d-483d-9cda-5372b1b0399f-kube-api-access-cwnqr\") pod \"postgres-7cdbf6dc44-9pfkh\" (UID: \"a51876bb-942d-483d-9cda-5372b1b0399f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.363642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.363559    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:00.662977 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:08:00.662948    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51876bb_942d_483d_9cda_5372b1b0399f.slice/crio-50e7f1485868793e8c2c43cca49a769ca6008cfe5ecd94e66827d1c49827f008 WatchSource:0}: Error finding container 50e7f1485868793e8c2c43cca49a769ca6008cfe5ecd94e66827d1c49827f008: Status 404 returned error can't find the container with id 50e7f1485868793e8c2c43cca49a769ca6008cfe5ecd94e66827d1c49827f008
Oct 30 03:08:00.663096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:00.662975    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"]
Oct 30 03:08:01.331170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:01.331135    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh" event={"ID":"a51876bb-942d-483d-9cda-5372b1b0399f","Type":"ContainerStarted","Data":"50e7f1485868793e8c2c43cca49a769ca6008cfe5ecd94e66827d1c49827f008"}
Oct 30 03:08:05.355878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:05.355788    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh" event={"ID":"a51876bb-942d-483d-9cda-5372b1b0399f","Type":"ContainerStarted","Data":"2e3beaaa3079cc061b7c2757055067741ebfdca5acb5b2625c1698d8e54f21b4"}
Oct 30 03:08:05.356274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:05.355937    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:05.412931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:05.412870    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh" podStartSLOduration=0.993136336 podStartE2EDuration="5.412854031s" totalImagesPullingTime="4.419717695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:05.411975453 +0000 UTC m=+313.840864890" watchObservedRunningTime="2026-10-30 03:08:05.412854031 +0000 UTC m=+313.841743450"
Oct 30 03:08:11.391804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:11.391771    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9pfkh"
Oct 30 03:08:16.228266 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:08:16.228229    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:08:23.421232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.421191    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"]
Oct 30 03:08:23.444238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.444202    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"]
Oct 30 03:08:23.444238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.444231    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.459725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.459700    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 03:08:23.459839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.459759    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nt4w2\""
Oct 30 03:08:23.472156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.472118    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 03:08:23.472303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.472279    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 03:08:23.472303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.472289    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 03:08:23.472415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.472331    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 03:08:23.573396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.573364    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.573396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.573398    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.573653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.573501    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhsc\" (UniqueName: \"kubernetes.io/projected/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-kube-api-access-8mhsc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.573653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.573546    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.674725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.674640    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.674725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.674678    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.674960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.674735    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhsc\" (UniqueName: \"kubernetes.io/projected/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-kube-api-access-8mhsc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.674960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.674766    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.675533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.675509    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.675733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.675713    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.677213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.677193    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.694320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.694295    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhsc\" (UniqueName: \"kubernetes.io/projected/f7a74cb9-1151-42d3-a91c-aafd6cfbc689-kube-api-access-8mhsc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ghfrc\" (UID: \"f7a74cb9-1151-42d3-a91c-aafd6cfbc689\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:23.762521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:23.762488    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:24.062688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:08:24.062652    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a74cb9_1151_42d3_a91c_aafd6cfbc689.slice/crio-ac6bc33e140962fac898a132ce1654f8087d69be76102e095efb025dc806ccf3 WatchSource:0}: Error finding container ac6bc33e140962fac898a132ce1654f8087d69be76102e095efb025dc806ccf3: Status 404 returned error can't find the container with id ac6bc33e140962fac898a132ce1654f8087d69be76102e095efb025dc806ccf3
Oct 30 03:08:24.063047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:24.063025    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"]
Oct 30 03:08:24.064733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:24.064717    4877 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:08:24.456791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:24.456705    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc" event={"ID":"f7a74cb9-1151-42d3-a91c-aafd6cfbc689","Type":"ContainerStarted","Data":"ac6bc33e140962fac898a132ce1654f8087d69be76102e095efb025dc806ccf3"}
Oct 30 03:08:37.363464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.363403    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-prfl7"]
Oct 30 03:08:37.366375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.366353    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.386486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.386456    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-prfl7"]
Oct 30 03:08:37.450406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.450319    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nl85p\""
Oct 30 03:08:37.450406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.450366    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 03:08:37.450406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.450391    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 03:08:37.518705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.518675    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0201f5d7-a014-4d69-983e-a25e14a48670-cert\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.518869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.518724    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtkq\" (UniqueName: \"kubernetes.io/projected/0201f5d7-a014-4d69-983e-a25e14a48670-kube-api-access-9gtkq\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.518869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.518842    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f5d7-a014-4d69-983e-a25e14a48670-metrics-certs\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.535239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.535203    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc" event={"ID":"f7a74cb9-1151-42d3-a91c-aafd6cfbc689","Type":"ContainerStarted","Data":"9994290e41e5bc8d2b79917a24135bd586014ecdf60104a8ae18e496092f9db2"}
Oct 30 03:08:37.535383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.535331    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:37.574944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.574890    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc" podStartSLOduration=1.4983548660000001 podStartE2EDuration="14.574876113s" totalImagesPullingTime="13.076521247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:37.573036728 +0000 UTC m=+346.001926174" watchObservedRunningTime="2026-10-30 03:08:37.574876113 +0000 UTC m=+346.003765536"
Oct 30 03:08:37.619568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.619535    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f5d7-a014-4d69-983e-a25e14a48670-metrics-certs\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.619735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.619711    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0201f5d7-a014-4d69-983e-a25e14a48670-cert\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.619825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.619804    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtkq\" (UniqueName: \"kubernetes.io/projected/0201f5d7-a014-4d69-983e-a25e14a48670-kube-api-access-9gtkq\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.622068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.622042    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f5d7-a014-4d69-983e-a25e14a48670-metrics-certs\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.622176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.622074    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0201f5d7-a014-4d69-983e-a25e14a48670-cert\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.638731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.638709    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtkq\" (UniqueName: \"kubernetes.io/projected/0201f5d7-a014-4d69-983e-a25e14a48670-kube-api-access-9gtkq\") pod \"odh-model-controller-769875db56-prfl7\" (UID: \"0201f5d7-a014-4d69-983e-a25e14a48670\") " pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.678907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.678881    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:37.983370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:37.983305    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-prfl7"]
Oct 30 03:08:37.983483 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:08:37.983307    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0201f5d7_a014_4d69_983e_a25e14a48670.slice/crio-41e87bd590de9f2d50bdf0a1d2f950a446a48c2c07caae2905a68941e0da5a92 WatchSource:0}: Error finding container 41e87bd590de9f2d50bdf0a1d2f950a446a48c2c07caae2905a68941e0da5a92: Status 404 returned error can't find the container with id 41e87bd590de9f2d50bdf0a1d2f950a446a48c2c07caae2905a68941e0da5a92
Oct 30 03:08:38.545098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:38.545048    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-prfl7" event={"ID":"0201f5d7-a014-4d69-983e-a25e14a48670","Type":"ContainerStarted","Data":"41e87bd590de9f2d50bdf0a1d2f950a446a48c2c07caae2905a68941e0da5a92"}
Oct 30 03:08:40.557193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:40.557163    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-prfl7" event={"ID":"0201f5d7-a014-4d69-983e-a25e14a48670","Type":"ContainerStarted","Data":"2749156600d7dab15a9c4643829c009dbf4ad56b83292f4d828f4a08cc706f94"}
Oct 30 03:08:40.557652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:40.557350    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:08:40.600238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:40.600192    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-prfl7" podStartSLOduration=1.113705062 podStartE2EDuration="3.600179811s" totalImagesPullingTime="2.486474749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:40.598573791 +0000 UTC m=+349.027463222" watchObservedRunningTime="2026-10-30 03:08:40.600179811 +0000 UTC m=+349.029069228"
Oct 30 03:08:46.229414 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:08:46.229379    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:08:48.551130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:48.551101    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ghfrc"
Oct 30 03:08:51.564657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:08:51.564628    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-prfl7"
Oct 30 03:09:02.161322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.161266    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hvmpm"]
Oct 30 03:09:02.180308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.180277    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:02.185170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.185130    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hvmpm"]
Oct 30 03:09:02.196885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.196859    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dd2cq\""
Oct 30 03:09:02.249530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.249496    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9kx\" (UniqueName: \"kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx\") pod \"authorino-6f6944bcdd-hvmpm\" (UID: \"a54f76a7-9980-4086-b716-18e616356ac3\") " pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:02.350301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.350262    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9kx\" (UniqueName: \"kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx\") pod \"authorino-6f6944bcdd-hvmpm\" (UID: \"a54f76a7-9980-4086-b716-18e616356ac3\") " pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:02.381343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.381303    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9kx\" (UniqueName: \"kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx\") pod \"authorino-6f6944bcdd-hvmpm\" (UID: \"a54f76a7-9980-4086-b716-18e616356ac3\") " pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:02.484587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.484504    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hvmpm"]
Oct 30 03:09:02.484751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.484739    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:02.527507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.526506    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-dhkcx"]
Oct 30 03:09:02.531851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.531827    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.557577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.550071    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-dhkcx"]
Oct 30 03:09:02.557577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.550715    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 03:09:02.585647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.585368    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-dhkcx"]
Oct 30 03:09:02.585647 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:09:02.585605    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rtt97 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-dhkcx" podUID="5700e2b0-cddd-4ee6-ba6d-62b6744e4935"
Oct 30 03:09:02.618347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.618305    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7858bd7855-v6qrn"]
Oct 30 03:09:02.621965 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.621943    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.639389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.639361    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7858bd7855-v6qrn"]
Oct 30 03:09:02.653057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.653027    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert\") pod \"authorino-6889cbd8c4-dhkcx\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") " pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.653226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.653063    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert\") pod \"authorino-7858bd7855-v6qrn\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") " pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.653226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.653097    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtt97\" (UniqueName: \"kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97\") pod \"authorino-6889cbd8c4-dhkcx\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") " pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.653346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.653274    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt55\" (UniqueName: \"kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55\") pod \"authorino-7858bd7855-v6qrn\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") " pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.671811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.671780    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.678929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.678896    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.753846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.753766    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt55\" (UniqueName: \"kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55\") pod \"authorino-7858bd7855-v6qrn\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") " pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.753846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.753813    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert\") pod \"authorino-6889cbd8c4-dhkcx\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") " pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.753846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.753836    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert\") pod \"authorino-7858bd7855-v6qrn\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") " pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.754126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.753857    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtt97\" (UniqueName: \"kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97\") pod \"authorino-6889cbd8c4-dhkcx\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") " pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.756332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.756308    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert\") pod \"authorino-7858bd7855-v6qrn\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") " pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.756481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.756458    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert\") pod \"authorino-6889cbd8c4-dhkcx\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") " pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.773971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.773943    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtt97\" (UniqueName: \"kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97\") pod \"authorino-6889cbd8c4-dhkcx\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") " pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:02.774132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.774108    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt55\" (UniqueName: \"kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55\") pod \"authorino-7858bd7855-v6qrn\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") " pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.789847 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:09:02.789818    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda54f76a7_9980_4086_b716_18e616356ac3.slice/crio-672a88031df645d771a52131449a0f9e49857c31bd0c93d011e8b8d7f37f1667 WatchSource:0}: Error finding container 672a88031df645d771a52131449a0f9e49857c31bd0c93d011e8b8d7f37f1667: Status 404 returned error can't find the container with id 672a88031df645d771a52131449a0f9e49857c31bd0c93d011e8b8d7f37f1667
Oct 30 03:09:02.790912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.790889    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hvmpm"]
Oct 30 03:09:02.854365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.854328    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert\" (UniqueName: \"kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert\") pod \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") "
Oct 30 03:09:02.854559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.854452    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97\" (UniqueName: \"kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97\") pod \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\" (UID: \"5700e2b0-cddd-4ee6-ba6d-62b6744e4935\") "
Oct 30 03:09:02.856531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.856506    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert" pod "5700e2b0-cddd-4ee6-ba6d-62b6744e4935" (UID: "5700e2b0-cddd-4ee6-ba6d-62b6744e4935"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:09:02.856609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.856534    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97" pod "5700e2b0-cddd-4ee6-ba6d-62b6744e4935" (UID: "5700e2b0-cddd-4ee6-ba6d-62b6744e4935"). InnerVolumeSpecName "kube-api-access-rtt97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:02.934934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.934901    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:09:02.955780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.955749    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtt97\" (UniqueName: \"kubernetes.io/projected/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-kube-api-access-rtt97\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:09:02.955889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:02.955785    4877 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5700e2b0-cddd-4ee6-ba6d-62b6744e4935-tls-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:09:03.234271 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:09:03.234238    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bd52423_1f58_41a0_b329_d86b15d218df.slice/crio-92e85f18da77b9f99ed1db33483b534855dec67d6bd23ce196319e15ed493265 WatchSource:0}: Error finding container 92e85f18da77b9f99ed1db33483b534855dec67d6bd23ce196319e15ed493265: Status 404 returned error can't find the container with id 92e85f18da77b9f99ed1db33483b534855dec67d6bd23ce196319e15ed493265
Oct 30 03:09:03.234625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.234575    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7858bd7855-v6qrn"]
Oct 30 03:09:03.680009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.679962    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm" event={"ID":"a54f76a7-9980-4086-b716-18e616356ac3","Type":"ContainerStarted","Data":"dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae"}
Oct 30 03:09:03.680179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.680026    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm" event={"ID":"a54f76a7-9980-4086-b716-18e616356ac3","Type":"ContainerStarted","Data":"672a88031df645d771a52131449a0f9e49857c31bd0c93d011e8b8d7f37f1667"}
Oct 30 03:09:03.680179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.680027    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm" podUID="a54f76a7-9980-4086-b716-18e616356ac3" containerName="authorino" containerID="cri-o://dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae" gracePeriod=30
Oct 30 03:09:03.682272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.682243    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-dhkcx"
Oct 30 03:09:03.685289 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.684370    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7858bd7855-v6qrn" event={"ID":"9bd52423-1f58-41a0-b329-d86b15d218df","Type":"ContainerStarted","Data":"55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678"}
Oct 30 03:09:03.685289 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.684418    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7858bd7855-v6qrn" event={"ID":"9bd52423-1f58-41a0-b329-d86b15d218df","Type":"ContainerStarted","Data":"92e85f18da77b9f99ed1db33483b534855dec67d6bd23ce196319e15ed493265"}
Oct 30 03:09:03.725491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.725440    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm" podStartSLOduration=1.219853391 podStartE2EDuration="1.72541504s" totalImagesPullingTime="505.561649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:03.724036752 +0000 UTC m=+372.152926169" watchObservedRunningTime="2026-10-30 03:09:03.72541504 +0000 UTC m=+372.154304458"
Oct 30 03:09:03.787988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.787957    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-dhkcx"]
Oct 30 03:09:03.792628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.791359    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-dhkcx"]
Oct 30 03:09:03.834272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.834075    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7858bd7855-v6qrn" podStartSLOduration=1.520894353 podStartE2EDuration="1.834054897s" totalImagesPullingTime="313.160544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:03.829863019 +0000 UTC m=+372.258752441" watchObservedRunningTime="2026-10-30 03:09:03.834054897 +0000 UTC m=+372.262944316"
Oct 30 03:09:03.955440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.955404    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:03.963910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.963800    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx\" (UniqueName: \"kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx\") pod \"a54f76a7-9980-4086-b716-18e616356ac3\" (UID: \"a54f76a7-9980-4086-b716-18e616356ac3\") "
Oct 30 03:09:03.967221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:03.967190    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx" pod "a54f76a7-9980-4086-b716-18e616356ac3" (UID: "a54f76a7-9980-4086-b716-18e616356ac3"). InnerVolumeSpecName "kube-api-access-6z9kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:09:04.065410 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.065322    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9kx\" (UniqueName: \"kubernetes.io/projected/a54f76a7-9980-4086-b716-18e616356ac3-kube-api-access-6z9kx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:09:04.688530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.688443    4877 generic.go:356] "Generic (PLEG): container finished" podID="d7badd71-7851-4ddb-b54b-86d2707078be" containerID="e0ac071e41e3523071fcd5a7a82e80808ef19baac3785315bd08faf6a2ff91d0" exitCode=0
Oct 30 03:09:04.688530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.688507    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" event={"ID":"d7badd71-7851-4ddb-b54b-86d2707078be","Type":"ContainerDied","Data":"e0ac071e41e3523071fcd5a7a82e80808ef19baac3785315bd08faf6a2ff91d0"}
Oct 30 03:09:04.689091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.688867    4877 scope.go:122] "RemoveContainer" containerID="e0ac071e41e3523071fcd5a7a82e80808ef19baac3785315bd08faf6a2ff91d0"
Oct 30 03:09:04.689984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.689961    4877 generic.go:356] "Generic (PLEG): container finished" podID="a54f76a7-9980-4086-b716-18e616356ac3" containerID="dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae" exitCode=0
Oct 30 03:09:04.690080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.689969    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:09:04.690080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.690001    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm"
Oct 30 03:09:04.690080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.690008    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm" event={"ID":"a54f76a7-9980-4086-b716-18e616356ac3","Type":"ContainerDied","Data":"dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae"}
Oct 30 03:09:04.690080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.690038    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hvmpm" event={"ID":"a54f76a7-9980-4086-b716-18e616356ac3","Type":"ContainerDied","Data":"672a88031df645d771a52131449a0f9e49857c31bd0c93d011e8b8d7f37f1667"}
Oct 30 03:09:04.690080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.690057    4877 scope.go:122] "RemoveContainer" containerID="dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae"
Oct 30 03:09:04.690080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.690055    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:09:04.690304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.690101    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:09:04.701320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.701301    4877 scope.go:122] "RemoveContainer" containerID="dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae"
Oct 30 03:09:04.701605 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:09:04.701588    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae\": container with ID starting with dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae not found: ID does not exist" containerID="dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae"
Oct 30 03:09:04.701670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.701616    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae"} err="failed to get container status \"dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae\": rpc error: code = NotFound desc = could not find container \"dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae\": container with ID starting with dce28642271049c1e84a671b810d3dc837d54b76fd25b6dbbb5be3df0a8f00ae not found: ID does not exist"
Oct 30 03:09:04.790768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.790724    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hvmpm"]
Oct 30 03:09:04.822651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:04.807306    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hvmpm"]
Oct 30 03:09:05.706474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:05.706437    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5700e2b0-cddd-4ee6-ba6d-62b6744e4935" path="/var/lib/kubelet/pods/5700e2b0-cddd-4ee6-ba6d-62b6744e4935/volumes"
Oct 30 03:09:05.706876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:05.706671    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54f76a7-9980-4086-b716-18e616356ac3" path="/var/lib/kubelet/pods/a54f76a7-9980-4086-b716-18e616356ac3/volumes"
Oct 30 03:09:05.706958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:05.706941    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" event={"ID":"d7badd71-7851-4ddb-b54b-86d2707078be","Type":"ContainerStarted","Data":"194b41bf8e625aa21603f5119e81b576347bfabb8fd091f56ba740a385cc32b9"}
Oct 30 03:09:05.741489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:05.741457    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:09:05.741665 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:05.741647    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 03:09:05.741744 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:05.741688    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 03:09:06.741380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:06.741345    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 03:09:06.741787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:06.741398    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 03:09:07.741795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:07.741760    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 03:09:07.742168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:07.741814    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 03:09:08.741511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:08.741475    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 03:09:08.741809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:08.741533    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 03:09:09.741575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:09.741544    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 03:09:09.741774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:09.741599    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 03:09:10.744897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:10.744867    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:09:10.745350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:10.745279    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:09:10.745934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:09:10.745916    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j"
Oct 30 03:09:16.227772 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:09:16.227734    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:09:46.227902 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:09:46.227866    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:10:16.227561 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:10:16.227524    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:10:46.229876 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:10:46.229837    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:10:56.508207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.508169    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"]
Oct 30 03:10:56.508727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.508566    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a54f76a7-9980-4086-b716-18e616356ac3" containerName="authorino"
Oct 30 03:10:56.508727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.508577    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54f76a7-9980-4086-b716-18e616356ac3" containerName="authorino"
Oct 30 03:10:56.508727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.508664    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54f76a7-9980-4086-b716-18e616356ac3" containerName="authorino"
Oct 30 03:10:56.511856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.511837    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.541167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.541131    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 03:10:56.541340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.541178    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9mjxw\""
Oct 30 03:10:56.541340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.541225    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 03:10:56.541340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.541300    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 03:10:56.543654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.543628    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"]
Oct 30 03:10:56.672287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.672248    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e4abcc-bfca-4458-8aed-8b4fca74e552-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.672496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.672294    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mp7\" (UniqueName: \"kubernetes.io/projected/c4e4abcc-bfca-4458-8aed-8b4fca74e552-kube-api-access-79mp7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.672496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.672351    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.672496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.672375    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.672496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.672392    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.672716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.672581    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.773659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.773625    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.773659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.773664    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.773906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.773704    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.773906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.773747    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e4abcc-bfca-4458-8aed-8b4fca74e552-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.773906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.773770    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mp7\" (UniqueName: \"kubernetes.io/projected/c4e4abcc-bfca-4458-8aed-8b4fca74e552-kube-api-access-79mp7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.773906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.773798    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.774161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.774089    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.774161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.774138    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.774264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.774185    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.776024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.776000    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e4abcc-bfca-4458-8aed-8b4fca74e552-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.776319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.776292    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e4abcc-bfca-4458-8aed-8b4fca74e552-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.792782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.792752    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mp7\" (UniqueName: \"kubernetes.io/projected/c4e4abcc-bfca-4458-8aed-8b4fca74e552-kube-api-access-79mp7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvg6r\" (UID: \"c4e4abcc-bfca-4458-8aed-8b4fca74e552\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:56.826237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:56.826200    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:10:57.136929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:57.136888    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"]
Oct 30 03:10:57.138350 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:10:57.138301    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e4abcc_bfca_4458_8aed_8b4fca74e552.slice/crio-14f70c8004a0e4a55ec0accd01ed07f991e672e954885eb07aff009a09270d04 WatchSource:0}: Error finding container 14f70c8004a0e4a55ec0accd01ed07f991e672e954885eb07aff009a09270d04: Status 404 returned error can't find the container with id 14f70c8004a0e4a55ec0accd01ed07f991e672e954885eb07aff009a09270d04
Oct 30 03:10:57.279101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:57.279049    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r" event={"ID":"c4e4abcc-bfca-4458-8aed-8b4fca74e552","Type":"ContainerStarted","Data":"14f70c8004a0e4a55ec0accd01ed07f991e672e954885eb07aff009a09270d04"}
Oct 30 03:10:58.551555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.551523    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"]
Oct 30 03:10:58.555762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.555738    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.575141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.575114    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"]
Oct 30 03:10:58.584360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.584333    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 03:10:58.597505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.597475    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.597964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.597697    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.598088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.597996    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41030117-c589-4881-8d38-fe2d36adbc70-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.598088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.598041    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.598088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.598063    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.598206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.598127    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgd6\" (UniqueName: \"kubernetes.io/projected/41030117-c589-4881-8d38-fe2d36adbc70-kube-api-access-5lgd6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699470    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699508    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41030117-c589-4881-8d38-fe2d36adbc70-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699536    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699651    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699696    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgd6\" (UniqueName: \"kubernetes.io/projected/41030117-c589-4881-8d38-fe2d36adbc70-kube-api-access-5lgd6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699809    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.699985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699959    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.700040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.699999    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.700136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.700107    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.701940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.701912    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41030117-c589-4881-8d38-fe2d36adbc70-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.702303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.702279    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41030117-c589-4881-8d38-fe2d36adbc70-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.718500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.718441    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgd6\" (UniqueName: \"kubernetes.io/projected/41030117-c589-4881-8d38-fe2d36adbc70-kube-api-access-5lgd6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j59xx\" (UID: \"41030117-c589-4881-8d38-fe2d36adbc70\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:58.871197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:58.871152    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:10:59.169690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:59.169660    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"]
Oct 30 03:10:59.171306 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:10:59.171280    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41030117_c589_4881_8d38_fe2d36adbc70.slice/crio-23d5346735fd3eebd72847aacb60166cf4a1d355b2b8c5ec3242f86de9ae691b WatchSource:0}: Error finding container 23d5346735fd3eebd72847aacb60166cf4a1d355b2b8c5ec3242f86de9ae691b: Status 404 returned error can't find the container with id 23d5346735fd3eebd72847aacb60166cf4a1d355b2b8c5ec3242f86de9ae691b
Oct 30 03:10:59.292194 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:59.292149    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r" event={"ID":"c4e4abcc-bfca-4458-8aed-8b4fca74e552","Type":"ContainerStarted","Data":"57db95ed311afe3b229b664757fd10a4fc00bb3c1bf1ade1fb3a2d39b3f00ea7"}
Oct 30 03:10:59.293617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:59.293586    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx" event={"ID":"41030117-c589-4881-8d38-fe2d36adbc70","Type":"ContainerStarted","Data":"23d5346735fd3eebd72847aacb60166cf4a1d355b2b8c5ec3242f86de9ae691b"}
Oct 30 03:10:59.332774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:59.332725    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r" podStartSLOduration=2.017772004 podStartE2EDuration="3.332711717s" totalImagesPullingTime="1.314939713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:59.331755603 +0000 UTC m=+487.760645032" watchObservedRunningTime="2026-10-30 03:10:59.332711717 +0000 UTC m=+487.761601158"
Oct 30 03:10:59.826612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:10:59.826584    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:11:00.303878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:00.303838    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx" event={"ID":"41030117-c589-4881-8d38-fe2d36adbc70","Type":"ContainerStarted","Data":"42be0771abdb46067f6a6ef14dcaefb80a8a35fb94cfe3b8ef9a6fa584634ae5"}
Oct 30 03:11:00.350692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:00.350622    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx" podStartSLOduration=2.172397321 podStartE2EDuration="2.350602679s" totalImagesPullingTime="178.205358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:00.347602004 +0000 UTC m=+488.776491450" watchObservedRunningTime="2026-10-30 03:11:00.350602679 +0000 UTC m=+488.779492099"
Oct 30 03:11:00.872129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:00.872095    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:11:02.065548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.065510    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"]
Oct 30 03:11:02.069950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.069928    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.089037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.089008    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"]
Oct 30 03:11:02.097922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.097895    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 03:11:02.135180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.135134    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtgc\" (UniqueName: \"kubernetes.io/projected/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-kube-api-access-fmtgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.135377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.135196    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.135377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.135349    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.135507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.135386    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.135507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.135409    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.135587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.135499    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.239448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.238916    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.239448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.238987    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.239448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.239047    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.239448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.239074    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.239448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.239111    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtgc\" (UniqueName: \"kubernetes.io/projected/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-kube-api-access-fmtgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.239448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.239156    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.241093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.240238    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.241093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.240519    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.241093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.240769    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.255286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.255252    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.256651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.256615    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.276448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.272259    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtgc\" (UniqueName: \"kubernetes.io/projected/b250ae5d-958a-4a5b-9eee-3936a1f2ec2a-kube-api-access-fmtgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5\" (UID: \"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.388405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.388307    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:02.713767 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:02.713736    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"]
Oct 30 03:11:02.715163 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:02.715138    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb250ae5d_958a_4a5b_9eee_3936a1f2ec2a.slice/crio-b4b3b5f52635829ce0093c99ad4329bf53c8b5f29e403a4bdc45240a38bbee4c WatchSource:0}: Error finding container b4b3b5f52635829ce0093c99ad4329bf53c8b5f29e403a4bdc45240a38bbee4c: Status 404 returned error can't find the container with id b4b3b5f52635829ce0093c99ad4329bf53c8b5f29e403a4bdc45240a38bbee4c
Oct 30 03:11:03.325903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:03.325858    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5" event={"ID":"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a","Type":"ContainerStarted","Data":"e9c6e5e3a2ad948109f6cdd2c0b28633ca3e25693a88148741ef35772d5a284f"}
Oct 30 03:11:03.326576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:03.326543    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5" event={"ID":"b250ae5d-958a-4a5b-9eee-3936a1f2ec2a","Type":"ContainerStarted","Data":"b4b3b5f52635829ce0093c99ad4329bf53c8b5f29e403a4bdc45240a38bbee4c"}
Oct 30 03:11:03.367468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:03.367402    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5" podStartSLOduration=1.191736629 podStartE2EDuration="1.367386481s" totalImagesPullingTime="175.649852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:03.366255827 +0000 UTC m=+491.795145293" watchObservedRunningTime="2026-10-30 03:11:03.367386481 +0000 UTC m=+491.796275900"
Oct 30 03:11:03.389302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:03.389250    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:05.758700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.757894    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"]
Oct 30 03:11:05.761630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.761604    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"]
Oct 30 03:11:05.761805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.761738    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:05.799936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.799907    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 03:11:05.930868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.930817    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:05.931052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.930896    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:05.931052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.930965    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:05.931170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.931081    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:05.931170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.931130    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e0b1b4-6014-40fc-a926-bc3780736c52-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:05.931170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:05.931161    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65b7r\" (UniqueName: \"kubernetes.io/projected/b8e0b1b4-6014-40fc-a926-bc3780736c52-kube-api-access-65b7r\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.031996    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032076    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032100    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032156    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032204    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e0b1b4-6014-40fc-a926-bc3780736c52-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032229    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65b7r\" (UniqueName: \"kubernetes.io/projected/b8e0b1b4-6014-40fc-a926-bc3780736c52-kube-api-access-65b7r\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032541    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.032893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.032864    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.033080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.033028    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.035720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.035655    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e0b1b4-6014-40fc-a926-bc3780736c52-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.035934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.035910    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e0b1b4-6014-40fc-a926-bc3780736c52-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.117550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.117513    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65b7r\" (UniqueName: \"kubernetes.io/projected/b8e0b1b4-6014-40fc-a926-bc3780736c52-kube-api-access-65b7r\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn\" (UID: \"b8e0b1b4-6014-40fc-a926-bc3780736c52\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.378045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.377958    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:06.688405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.688374    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"]
Oct 30 03:11:06.690031 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:06.689991    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e0b1b4_6014_40fc_a926_bc3780736c52.slice/crio-1fdc25a99c2ce4ba872c7a7fc7ced98a8837932ad9bc116a78c5ead3cd0fdc57 WatchSource:0}: Error finding container 1fdc25a99c2ce4ba872c7a7fc7ced98a8837932ad9bc116a78c5ead3cd0fdc57: Status 404 returned error can't find the container with id 1fdc25a99c2ce4ba872c7a7fc7ced98a8837932ad9bc116a78c5ead3cd0fdc57
Oct 30 03:11:06.826858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.826815    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:11:06.839223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:06.839199    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:11:07.354576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:07.354543    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn" event={"ID":"b8e0b1b4-6014-40fc-a926-bc3780736c52","Type":"ContainerStarted","Data":"7b3458ae557b2394ca260bf4bf7c3688cc2846c7678be4e30fd842e4765a21a3"}
Oct 30 03:11:07.354787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:07.354597    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn" event={"ID":"b8e0b1b4-6014-40fc-a926-bc3780736c52","Type":"ContainerStarted","Data":"1fdc25a99c2ce4ba872c7a7fc7ced98a8837932ad9bc116a78c5ead3cd0fdc57"}
Oct 30 03:11:07.366043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:07.366010    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvg6r"
Oct 30 03:11:07.378850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:07.378823    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:07.423924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:07.423871    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn" podStartSLOduration=2.234704699 podStartE2EDuration="2.423858047s" totalImagesPullingTime="189.153348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:07.421668657 +0000 UTC m=+495.850558075" watchObservedRunningTime="2026-10-30 03:11:07.423858047 +0000 UTC m=+495.852747464"
Oct 30 03:11:08.871815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:08.871786    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:11:08.886097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:08.886065    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:11:09.381501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:09.381459    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j59xx"
Oct 30 03:11:12.388517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:12.388475    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:12.402057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:12.402023    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:13.402847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:13.402817    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5"
Oct 30 03:11:16.227841 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:11:16.227807    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:11:16.378753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:16.378725    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:16.392201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:16.392176    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:16.425908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:16.425884    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn"
Oct 30 03:11:17.345975 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.345936    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"]
Oct 30 03:11:17.352158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.352129    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.398294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.398255    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"]
Oct 30 03:11:17.446523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446481    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446601    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkklm\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446632    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446655    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446712    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446835    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446875    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446951    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.446986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.446980    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547478    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547580    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkklm\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547613    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547650    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547680    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547722    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.547974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.547879    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548033    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548110    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548245    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548262    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548292    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548586    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.548863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.548845    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.549986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.549967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.550119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.550101    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.567309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.567287    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.567309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.567302    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkklm\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-8cf96\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:17.670629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:17.670537    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:18.006683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.006648    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"]
Oct 30 03:11:18.008078 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:18.008048    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce76e7f_a3b1_4099_bc03_1c2d5e33d935.slice/crio-190604e77fa7d6cdd36fbb20d4a5de6f687d748a8acb6ec7c3c0c12922596196 WatchSource:0}: Error finding container 190604e77fa7d6cdd36fbb20d4a5de6f687d748a8acb6ec7c3c0c12922596196: Status 404 returned error can't find the container with id 190604e77fa7d6cdd36fbb20d4a5de6f687d748a8acb6ec7c3c0c12922596196
Oct 30 03:11:18.010296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.010264    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:18.010385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.010341    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:18.010486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.010387    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:18.434227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.434186    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" event={"ID":"cce76e7f-a3b1-4099-bc03-1c2d5e33d935","Type":"ContainerStarted","Data":"6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5"}
Oct 30 03:11:18.435916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.435883    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" event={"ID":"cce76e7f-a3b1-4099-bc03-1c2d5e33d935","Type":"ContainerStarted","Data":"190604e77fa7d6cdd36fbb20d4a5de6f687d748a8acb6ec7c3c0c12922596196"}
Oct 30 03:11:18.502228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.502167    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podStartSLOduration=1.5021487759999999 podStartE2EDuration="1.502148776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:18.501814873 +0000 UTC m=+506.930704294" watchObservedRunningTime="2026-10-30 03:11:18.502148776 +0000 UTC m=+506.931038195"
Oct 30 03:11:18.671706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:18.671676    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:19.671996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:19.671963    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-8cf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 03:11:19.672370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:19.672023    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 03:11:20.672009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:20.671971    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-8cf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 03:11:20.672393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:20.672042    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 03:11:21.671350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.671314    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-8cf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 03:11:21.671550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.671368    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 03:11:21.707218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.707177    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66b66c856-5qx55"]
Oct 30 03:11:21.711793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.711770    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:21.721214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.721189    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b66c856-5qx55"]
Oct 30 03:11:21.808559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.808519    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f34c49c1-2d78-4b79-8505-f68314e4d4b9-tls-cert\") pod \"authorino-66b66c856-5qx55\" (UID: \"f34c49c1-2d78-4b79-8505-f68314e4d4b9\") " pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:21.808718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.808586    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr6nw\" (UniqueName: \"kubernetes.io/projected/f34c49c1-2d78-4b79-8505-f68314e4d4b9-kube-api-access-hr6nw\") pod \"authorino-66b66c856-5qx55\" (UID: \"f34c49c1-2d78-4b79-8505-f68314e4d4b9\") " pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:21.909960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.909926    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f34c49c1-2d78-4b79-8505-f68314e4d4b9-tls-cert\") pod \"authorino-66b66c856-5qx55\" (UID: \"f34c49c1-2d78-4b79-8505-f68314e4d4b9\") " pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:21.910108 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.909986    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr6nw\" (UniqueName: \"kubernetes.io/projected/f34c49c1-2d78-4b79-8505-f68314e4d4b9-kube-api-access-hr6nw\") pod \"authorino-66b66c856-5qx55\" (UID: \"f34c49c1-2d78-4b79-8505-f68314e4d4b9\") " pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:21.912561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.912536    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f34c49c1-2d78-4b79-8505-f68314e4d4b9-tls-cert\") pod \"authorino-66b66c856-5qx55\" (UID: \"f34c49c1-2d78-4b79-8505-f68314e4d4b9\") " pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:21.928921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:21.928847    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr6nw\" (UniqueName: \"kubernetes.io/projected/f34c49c1-2d78-4b79-8505-f68314e4d4b9-kube-api-access-hr6nw\") pod \"authorino-66b66c856-5qx55\" (UID: \"f34c49c1-2d78-4b79-8505-f68314e4d4b9\") " pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:22.025634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:22.025597    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b66c856-5qx55"
Oct 30 03:11:22.360178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:22.360145    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b66c856-5qx55"]
Oct 30 03:11:22.361715 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:22.361677    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf34c49c1_2d78_4b79_8505_f68314e4d4b9.slice/crio-5fc0f70494fafc584d1c96c25916683d3ff90688faee14b6ca1d2f41ad116026 WatchSource:0}: Error finding container 5fc0f70494fafc584d1c96c25916683d3ff90688faee14b6ca1d2f41ad116026: Status 404 returned error can't find the container with id 5fc0f70494fafc584d1c96c25916683d3ff90688faee14b6ca1d2f41ad116026
Oct 30 03:11:22.455327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:22.455286    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b66c856-5qx55" event={"ID":"f34c49c1-2d78-4b79-8505-f68314e4d4b9","Type":"ContainerStarted","Data":"5fc0f70494fafc584d1c96c25916683d3ff90688faee14b6ca1d2f41ad116026"}
Oct 30 03:11:22.671467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:22.671354    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-8cf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 03:11:22.671628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:22.671459    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 03:11:23.462465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.462411    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b66c856-5qx55" event={"ID":"f34c49c1-2d78-4b79-8505-f68314e4d4b9","Type":"ContainerStarted","Data":"e841bf35214d501aca68ac699d06a718f3917c33a27d7362ca52c71c7ae04369"}
Oct 30 03:11:23.525993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.525923    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66b66c856-5qx55" podStartSLOduration=1.895242302 podStartE2EDuration="2.525904055s" totalImagesPullingTime="630.661753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:23.525554327 +0000 UTC m=+511.954443745" watchObservedRunningTime="2026-10-30 03:11:23.525904055 +0000 UTC m=+511.954793477"
Oct 30 03:11:23.557009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.556973    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7858bd7855-v6qrn"]
Oct 30 03:11:23.557240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.557207    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7858bd7855-v6qrn" podUID="9bd52423-1f58-41a0-b329-d86b15d218df" containerName="authorino" containerID="cri-o://55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678" gracePeriod=30
Oct 30 03:11:23.671343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.671309    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-8cf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 03:11:23.671547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.671368    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 03:11:23.869478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.869452    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:11:23.933012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.932971    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert\" (UniqueName: \"kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert\") pod \"9bd52423-1f58-41a0-b329-d86b15d218df\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") "
Oct 30 03:11:23.933636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.933616    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55\" (UniqueName: \"kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55\") pod \"9bd52423-1f58-41a0-b329-d86b15d218df\" (UID: \"9bd52423-1f58-41a0-b329-d86b15d218df\") "
Oct 30 03:11:23.936283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.936248    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55" pod "9bd52423-1f58-41a0-b329-d86b15d218df" (UID: "9bd52423-1f58-41a0-b329-d86b15d218df"). InnerVolumeSpecName "kube-api-access-vrt55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:11:23.953079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:23.953029    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert" pod "9bd52423-1f58-41a0-b329-d86b15d218df" (UID: "9bd52423-1f58-41a0-b329-d86b15d218df"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:11:24.035819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.035752    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt55\" (UniqueName: \"kubernetes.io/projected/9bd52423-1f58-41a0-b329-d86b15d218df-kube-api-access-vrt55\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:11:24.035819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.035786    4877 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9bd52423-1f58-41a0-b329-d86b15d218df-tls-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:11:24.471969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.471851    4877 generic.go:356] "Generic (PLEG): container finished" podID="9bd52423-1f58-41a0-b329-d86b15d218df" containerID="55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678" exitCode=0
Oct 30 03:11:24.472706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.472688    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7858bd7855-v6qrn"
Oct 30 03:11:24.474296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.474071    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7858bd7855-v6qrn" event={"ID":"9bd52423-1f58-41a0-b329-d86b15d218df","Type":"ContainerDied","Data":"55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678"}
Oct 30 03:11:24.474296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.474132    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7858bd7855-v6qrn" event={"ID":"9bd52423-1f58-41a0-b329-d86b15d218df","Type":"ContainerDied","Data":"92e85f18da77b9f99ed1db33483b534855dec67d6bd23ce196319e15ed493265"}
Oct 30 03:11:24.474296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.474157    4877 scope.go:122] "RemoveContainer" containerID="55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678"
Oct 30 03:11:24.486763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.486737    4877 scope.go:122] "RemoveContainer" containerID="55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678"
Oct 30 03:11:24.487094 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:11:24.487075    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678\": container with ID starting with 55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678 not found: ID does not exist" containerID="55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678"
Oct 30 03:11:24.487150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.487105    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678"} err="failed to get container status \"55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678\": rpc error: code = NotFound desc = could not find container \"55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678\": container with ID starting with 55454ec758f13b3640c23337361cfff00473a244ba98bc9f62d648b02ddae678 not found: ID does not exist"
Oct 30 03:11:24.531956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.531922    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7858bd7855-v6qrn"]
Oct 30 03:11:24.543539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.543511    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7858bd7855-v6qrn"]
Oct 30 03:11:24.671685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.671644    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-8cf96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 03:11:24.671882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:24.671700    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 03:11:25.705446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:25.702400    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd52423-1f58-41a0-b329-d86b15d218df" path="/var/lib/kubelet/pods/9bd52423-1f58-41a0-b329-d86b15d218df/volumes"
Oct 30 03:11:26.486608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:26.486577    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:26.487829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:26.487796    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:26.489004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:26.488969    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:11:46.227793 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:11:46.227754    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:11:56.440488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.440454    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"]
Oct 30 03:11:56.441145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.440908    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9bd52423-1f58-41a0-b329-d86b15d218df" containerName="authorino"
Oct 30 03:11:56.441145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.440923    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd52423-1f58-41a0-b329-d86b15d218df" containerName="authorino"
Oct 30 03:11:56.441145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.441038    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd52423-1f58-41a0-b329-d86b15d218df" containerName="authorino"
Oct 30 03:11:56.445520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.445495    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463123    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a788fe69-openshift-default-dockercfg-jzd27\""
Oct 30 03:11:56.463285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463250    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463290    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463350    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463393    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2k6\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463483    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463532    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463561    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463591    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.463738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.463716    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.475119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.475093    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"]
Oct 30 03:11:56.565012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.564972    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565060    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565129    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565158    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565181    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565223    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2k6\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565264    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565331    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565359    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565675    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565689    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.565942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565912    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.566081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.565935    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.566160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.566137    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.568296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.568273    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.568795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.568776    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.594743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.594710    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.596071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.596037    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2k6\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6\") pod \"e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") " pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.759359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.759224    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:56.816691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.812912    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"]
Oct 30 03:11:56.823872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.823846    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.841985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.839684    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-acace923-gw-openshift-default-dockercfg-w4fqr\""
Oct 30 03:11:56.841985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.840694    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"]
Oct 30 03:11:56.867986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.867953    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7j75\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868003    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868032    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868075    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868104    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868125    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868184    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868212    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.868418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.868269    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.969604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969575    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.969796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969611    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.969796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969635    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.969796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969693    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.969796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969720    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.969796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969771    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969847    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7j75\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.969883    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.970143    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.970165    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.970187    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.970367    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.970538    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.970698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.970683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.971957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.971931    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.972192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.972176    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.989606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.989568    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:56.989769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:56.989621    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7j75\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75\") pod \"e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") " pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:57.082103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.082072    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"]
Oct 30 03:11:57.083794 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:57.083765    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733630f7_7e30_4437_bcbe_b13ad98034e4.slice/crio-168db24a95e33bbafbc3c0b3bb4403552c5b261588ef5f5b6095c19450ebce46 WatchSource:0}: Error finding container 168db24a95e33bbafbc3c0b3bb4403552c5b261588ef5f5b6095c19450ebce46: Status 404 returned error can't find the container with id 168db24a95e33bbafbc3c0b3bb4403552c5b261588ef5f5b6095c19450ebce46
Oct 30 03:11:57.085897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.085860    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:57.085973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.085933    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:57.085973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.085960    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:57.152939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.152912    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:57.472265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.472238    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"]
Oct 30 03:11:57.473794 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:57.473767    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522a99c5_2000_4528_a092_bb00174b73f6.slice/crio-e72572fb4a73e050119e7cc22c310589c4b64a6c8003cea74055835dc71aea1f WatchSource:0}: Error finding container e72572fb4a73e050119e7cc22c310589c4b64a6c8003cea74055835dc71aea1f: Status 404 returned error can't find the container with id e72572fb4a73e050119e7cc22c310589c4b64a6c8003cea74055835dc71aea1f
Oct 30 03:11:57.476005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.475974    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:57.476096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.476039    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:57.476096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.476089    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:57.666997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.666965    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" event={"ID":"733630f7-7e30-4437-bcbe-b13ad98034e4","Type":"ContainerStarted","Data":"12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"}
Oct 30 03:11:57.667191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.667020    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" event={"ID":"733630f7-7e30-4437-bcbe-b13ad98034e4","Type":"ContainerStarted","Data":"168db24a95e33bbafbc3c0b3bb4403552c5b261588ef5f5b6095c19450ebce46"}
Oct 30 03:11:57.668611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.668586    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" event={"ID":"522a99c5-2000-4528-a092-bb00174b73f6","Type":"ContainerStarted","Data":"4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"}
Oct 30 03:11:57.668718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.668630    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" event={"ID":"522a99c5-2000-4528-a092-bb00174b73f6","Type":"ContainerStarted","Data":"e72572fb4a73e050119e7cc22c310589c4b64a6c8003cea74055835dc71aea1f"}
Oct 30 03:11:57.709914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.709850    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podStartSLOduration=1.709832815 podStartE2EDuration="1.709832815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:57.708614199 +0000 UTC m=+546.137503640" watchObservedRunningTime="2026-10-30 03:11:57.709832815 +0000 UTC m=+546.138722233"
Oct 30 03:11:57.748792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.748687    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podStartSLOduration=1.7486730050000001 podStartE2EDuration="1.748673005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:57.748305611 +0000 UTC m=+546.177195039" watchObservedRunningTime="2026-10-30 03:11:57.748673005 +0000 UTC m=+546.177562424"
Oct 30 03:11:57.760390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:57.760361    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:58.059674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.059640    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"]
Oct 30 03:11:58.063397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.063375    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.078113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.078088    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-421ef828-openshift-default-dockercfg-g79nj\""
Oct 30 03:11:58.082438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082398    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082551    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082592    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdpk\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082657    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082713    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082746    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082821    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.082998 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.082892    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.086445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.086402    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"]
Oct 30 03:11:58.153373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.153340    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:58.158357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.158333    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.183700    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.183888    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.183933    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184048    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184120    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdpk\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184150    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184181    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184211    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184277    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.184810    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.186885    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.187452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.187141    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.188732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.188709    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.189520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.189341    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.194333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.194307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.194831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.194811    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.210157 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.210125    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.211349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.211321    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdpk\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk\") pod \"e2e-ait-421ef828-openshift-default-59759b6669-cnhw2\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") " pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.376891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.376813    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:11:58.675103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.675016    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:58.676000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.675976    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:11:58.715383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.715362    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"]
Oct 30 03:11:58.718233 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:11:58.718206    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fa12d0_de53_4f30_ba56_dd3f8ea1383d.slice/crio-152f94fefb3734c912e695e27e769ca83dbb5cc7b7bf8b298412a86d2b1bda5a WatchSource:0}: Error finding container 152f94fefb3734c912e695e27e769ca83dbb5cc7b7bf8b298412a86d2b1bda5a: Status 404 returned error can't find the container with id 152f94fefb3734c912e695e27e769ca83dbb5cc7b7bf8b298412a86d2b1bda5a
Oct 30 03:11:58.726126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.726093    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:58.726255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.726175    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:58.726255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.726244    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:11:58.764384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:58.764356    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:59.683556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:59.683518    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" event={"ID":"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d","Type":"ContainerStarted","Data":"1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"}
Oct 30 03:11:59.684078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:59.683576    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" event={"ID":"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d","Type":"ContainerStarted","Data":"152f94fefb3734c912e695e27e769ca83dbb5cc7b7bf8b298412a86d2b1bda5a"}
Oct 30 03:11:59.684078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:59.683761    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:59.684881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:59.684857    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:11:59.732992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:11:59.732942    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podStartSLOduration=1.7329282190000002 podStartE2EDuration="1.732928219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:59.731749249 +0000 UTC m=+548.160638667" watchObservedRunningTime="2026-10-30 03:11:59.732928219 +0000 UTC m=+548.161817636"
Oct 30 03:12:00.377783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.377744    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:00.382847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.382819    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:00.690161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.690064    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:00.690811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.690793    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:00.808324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.808286    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"]
Oct 30 03:12:00.812062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.812038    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:00.834819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.834789    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"]
Oct 30 03:12:00.855577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.855549    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-acace923\""
Oct 30 03:12:00.859358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.859332    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-acace923-dockercfg-7vlzc\""
Oct 30 03:12:00.913091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.913046    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:00.913290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:00.913197    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ht6\" (UniqueName: \"kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6\") pod \"payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:01.014072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.013977    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ht6\" (UniqueName: \"kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6\") pod \"payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:01.014072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.014055    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:01.014950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.014926    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:01.032943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.032912    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ht6\" (UniqueName: \"kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6\") pod \"payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:01.125843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.125807    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:01.443786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.443759    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"]
Oct 30 03:12:01.444507 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:01.444482    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921202ff_7245_4756_b6e6_7ef958491b6e.slice/crio-af328aad52b285afd0a239925c75ac8399cd1e4f32cc87e1bb1a42c6eda4651e WatchSource:0}: Error finding container af328aad52b285afd0a239925c75ac8399cd1e4f32cc87e1bb1a42c6eda4651e: Status 404 returned error can't find the container with id af328aad52b285afd0a239925c75ac8399cd1e4f32cc87e1bb1a42c6eda4651e
Oct 30 03:12:01.697401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:01.697324    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26" event={"ID":"921202ff-7245-4756-b6e6-7ef958491b6e","Type":"ContainerStarted","Data":"af328aad52b285afd0a239925c75ac8399cd1e4f32cc87e1bb1a42c6eda4651e"}
Oct 30 03:12:04.721602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.721286    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26" event={"ID":"921202ff-7245-4756-b6e6-7ef958491b6e","Type":"ContainerStarted","Data":"769fcc60699a9a88982567071f08815a9a33e541480fc3babb0e334b87d26a09"}
Oct 30 03:12:04.722176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.722155    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:04.768627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.768519    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26" podStartSLOduration=2.47578765 podStartE2EDuration="4.768504331s" totalImagesPullingTime="2.292716681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:04.768276831 +0000 UTC m=+553.197166265" watchObservedRunningTime="2026-10-30 03:12:04.768504331 +0000 UTC m=+553.197393748"
Oct 30 03:12:04.866495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.866449    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"]
Oct 30 03:12:04.871051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.871027    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.907731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.907689    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3f9206f0-openshift-default-dockercfg-tgs44\""
Oct 30 03:12:04.909676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.909647    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"]
Oct 30 03:12:04.959159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959084    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959135    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkxx\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959246    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959403    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959449    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959489    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959584    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959648    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:04.959756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:04.959683    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.029911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.029882    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"]
Oct 30 03:12:05.033530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.033514    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.054474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.054412    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c7c56c3d\""
Oct 30 03:12:05.054692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.054596    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c7c56c3d-dockercfg-gb7rd\""
Oct 30 03:12:05.055632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.055604    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"]
Oct 30 03:12:05.061244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061211    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061283    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061335    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061365    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tq8x\" (UniqueName: \"kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x\") pod \"payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") " pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.061509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061401    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061491    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061544    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkxx\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061579    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") " pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.061678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061609    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061621    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061613    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061760    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.061936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.061787    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.062216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.062192    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.062414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.062369    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.062950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.062927    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.064008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.063983    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.064755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.064733    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.109255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.109216    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.110020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.109990    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkxx\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx\") pod \"e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") " pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.167925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.162484    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tq8x\" (UniqueName: \"kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x\") pod \"payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") " pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.167925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.162551    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") " pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.167925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.163400    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") " pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.187467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.186198    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tq8x\" (UniqueName: \"kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x\") pod \"payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") " pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.218474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.216670    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:05.347156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.347036    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:05.546071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.546037    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"]
Oct 30 03:12:05.547350 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:05.547312    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86a7659_10d5_4066_b558_c4db7c662362.slice/crio-5cc1bcad8bf847d442b11c595c027fb38d1f0bcd25c68808cfebdfd55bf77d64 WatchSource:0}: Error finding container 5cc1bcad8bf847d442b11c595c027fb38d1f0bcd25c68808cfebdfd55bf77d64: Status 404 returned error can't find the container with id 5cc1bcad8bf847d442b11c595c027fb38d1f0bcd25c68808cfebdfd55bf77d64
Oct 30 03:12:05.549669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.549639    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:05.549774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.549694    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:05.549774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.549725    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:05.733259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.733233    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"]
Oct 30 03:12:05.737808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.737774    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" event={"ID":"f86a7659-10d5-4066-b558-c4db7c662362","Type":"ContainerStarted","Data":"904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"}
Oct 30 03:12:05.737932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.737832    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" event={"ID":"f86a7659-10d5-4066-b558-c4db7c662362","Type":"ContainerStarted","Data":"5cc1bcad8bf847d442b11c595c027fb38d1f0bcd25c68808cfebdfd55bf77d64"}
Oct 30 03:12:05.741627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.741601    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9" event={"ID":"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f","Type":"ContainerStarted","Data":"84a7a25ed53af5a0914e89b36e56047a2d40b45f1876ed684770e78c20b6f2a4"}
Oct 30 03:12:05.970583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:05.948071    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podStartSLOduration=1.948047721 podStartE2EDuration="1.948047721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:05.797315144 +0000 UTC m=+554.226204560" watchObservedRunningTime="2026-10-30 03:12:05.948047721 +0000 UTC m=+554.376937139"
Oct 30 03:12:06.015074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.015034    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"]
Oct 30 03:12:06.031523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.031323    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"]
Oct 30 03:12:06.031523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.031481    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.046279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.046241    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-421ef828\""
Oct 30 03:12:06.051583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.051559    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-421ef828-dockercfg-bqvwb\""
Oct 30 03:12:06.174703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.174668    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcrj\" (UniqueName: \"kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj\") pod \"payload-processing-e2e-ait-421ef828-8568565666-qmh9h\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") " pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.174886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.174714    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume\") pod \"payload-processing-e2e-ait-421ef828-8568565666-qmh9h\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") " pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.219839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.219800    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:06.225293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.225216    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:06.306072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.306026    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcrj\" (UniqueName: \"kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj\") pod \"payload-processing-e2e-ait-421ef828-8568565666-qmh9h\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") " pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.306274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.306119    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume\") pod \"payload-processing-e2e-ait-421ef828-8568565666-qmh9h\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") " pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.307159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.307127    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume\") pod \"payload-processing-e2e-ait-421ef828-8568565666-qmh9h\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") " pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.329340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.329308    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcrj\" (UniqueName: \"kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj\") pod \"payload-processing-e2e-ait-421ef828-8568565666-qmh9h\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") " pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.357459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.357412    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:06.717265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.717235    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"]
Oct 30 03:12:06.720092 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:06.720064    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7902e666_5bea_4428_bd6a_76399e41792c.slice/crio-c6550f55e4c9c335aa7b3babfa2f8c123afa2140c50ef496e01f24b193a3e3b2 WatchSource:0}: Error finding container c6550f55e4c9c335aa7b3babfa2f8c123afa2140c50ef496e01f24b193a3e3b2: Status 404 returned error can't find the container with id c6550f55e4c9c335aa7b3babfa2f8c123afa2140c50ef496e01f24b193a3e3b2
Oct 30 03:12:06.748155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.748114    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9" event={"ID":"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f","Type":"ContainerStarted","Data":"7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53"}
Oct 30 03:12:06.748584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.748192    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:06.749750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.749722    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h" event={"ID":"7902e666-5bea-4428-bd6a-76399e41792c","Type":"ContainerStarted","Data":"c6550f55e4c9c335aa7b3babfa2f8c123afa2140c50ef496e01f24b193a3e3b2"}
Oct 30 03:12:06.749959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.749935    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:06.750947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.750926    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:06.846528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:06.846465    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9" podStartSLOduration=1.846413769 podStartE2EDuration="1.846413769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:06.804130574 +0000 UTC m=+555.233019996" watchObservedRunningTime="2026-10-30 03:12:06.846413769 +0000 UTC m=+555.275303185"
Oct 30 03:12:07.757855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:07.757814    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h" event={"ID":"7902e666-5bea-4428-bd6a-76399e41792c","Type":"ContainerStarted","Data":"720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c"}
Oct 30 03:12:07.758272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:07.758026    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:07.800254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:07.800195    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h" podStartSLOduration=2.800174102 podStartE2EDuration="2.800174102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:07.799862873 +0000 UTC m=+556.228752315" watchObservedRunningTime="2026-10-30 03:12:07.800174102 +0000 UTC m=+556.229063521"
Oct 30 03:12:10.447084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.446990    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"]
Oct 30 03:12:10.452675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.452646    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.459500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.459469    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.459631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.459519    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsf5q\" (UniqueName: \"kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q\") pod \"payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.466277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.466253    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"]
Oct 30 03:12:10.474940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.474918    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3f9206f0\""
Oct 30 03:12:10.474940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.474932    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3f9206f0-dockercfg-gjfwk\""
Oct 30 03:12:10.553131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.550635    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"]
Oct 30 03:12:10.555907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.555863    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.560166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.560136    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrs6\" (UniqueName: \"kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6\") pod \"payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") " pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.560320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.560295    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.560470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.560335    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsf5q\" (UniqueName: \"kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q\") pod \"payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.560470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.560375    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume\") pod \"payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") " pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.561457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.561392    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.575409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.575379    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"]
Oct 30 03:12:10.589211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.589178    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsf5q\" (UniqueName: \"kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q\") pod \"payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.661216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.661173    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume\") pod \"payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") " pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.661376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.661266    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrs6\" (UniqueName: \"kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6\") pod \"payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") " pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.661923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.661905    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume\") pod \"payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") " pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.712495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.712326    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrs6\" (UniqueName: \"kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6\") pod \"payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") " pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:10.768115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.768079    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:10.874555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:10.874322    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:11.106226 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:11.106189    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1514bc1c_9d56_48a2_864f_8640bf315dda.slice/crio-fe7a1f1cadcf6af28ace76a0c0802a436afc30c8569eaf8043be1738ff200424 WatchSource:0}: Error finding container fe7a1f1cadcf6af28ace76a0c0802a436afc30c8569eaf8043be1738ff200424: Status 404 returned error can't find the container with id fe7a1f1cadcf6af28ace76a0c0802a436afc30c8569eaf8043be1738ff200424
Oct 30 03:12:11.107024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.106997    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"]
Oct 30 03:12:11.210260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.210221    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"]
Oct 30 03:12:11.219236 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:11.219202    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a58d2f_2944_4520_a3b0_e6ae65984f2b.slice/crio-cef733607875d2929f6b3ac49ce4b4fd7b4f726cf0cbcf0b288e2493f315340d WatchSource:0}: Error finding container cef733607875d2929f6b3ac49ce4b4fd7b4f726cf0cbcf0b288e2493f315340d: Status 404 returned error can't find the container with id cef733607875d2929f6b3ac49ce4b4fd7b4f726cf0cbcf0b288e2493f315340d
Oct 30 03:12:11.800744 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.800697    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl" event={"ID":"34a58d2f-2944-4520-a3b0-e6ae65984f2b","Type":"ContainerStarted","Data":"e7e521ebb13c57679a2d531434ac44afd0472f78cea20f5124e25087999b3d4c"}
Oct 30 03:12:11.801280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.800770    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl" event={"ID":"34a58d2f-2944-4520-a3b0-e6ae65984f2b","Type":"ContainerStarted","Data":"cef733607875d2929f6b3ac49ce4b4fd7b4f726cf0cbcf0b288e2493f315340d"}
Oct 30 03:12:11.806799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.806766    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:11.809482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.808872    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8" event={"ID":"1514bc1c-9d56-48a2-864f-8640bf315dda","Type":"ContainerStarted","Data":"b9ffd31869a7677a7f370c66213059798d422a9829fd8f2b2746408eb21a815e"}
Oct 30 03:12:11.809482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.808921    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8" event={"ID":"1514bc1c-9d56-48a2-864f-8640bf315dda","Type":"ContainerStarted","Data":"fe7a1f1cadcf6af28ace76a0c0802a436afc30c8569eaf8043be1738ff200424"}
Oct 30 03:12:11.809482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.808958    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:11.848076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.848016    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl" podStartSLOduration=1.848001655 podStartE2EDuration="1.848001655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:11.845011433 +0000 UTC m=+560.273900875" watchObservedRunningTime="2026-10-30 03:12:11.848001655 +0000 UTC m=+560.276891073"
Oct 30 03:12:11.890446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:11.883210    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8" podStartSLOduration=1.8831923640000001 podStartE2EDuration="1.883192364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:11.882845704 +0000 UTC m=+560.311735147" watchObservedRunningTime="2026-10-30 03:12:11.883192364 +0000 UTC m=+560.312081783"
Oct 30 03:12:14.393743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.372777    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:48012->10.133.0.57:15021: read: connection reset by peer" start-of-body=
Oct 30 03:12:14.393743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.372872    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:48012->10.133.0.57:15021: read: connection reset by peer"
Oct 30 03:12:14.542949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.520698    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"]
Oct 30 03:12:14.542949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.541332    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.560977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.559243    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"]
Oct 30 03:12:14.629150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.629074    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:12:14.629369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.629350    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:12:14.641935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.641904    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.642112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.641942    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.642112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.641973    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65cr\" (UniqueName: \"kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.743641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.743599    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.743641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.743635    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.743894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.743661    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65cr\" (UniqueName: \"kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.744311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.744284    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.746083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.746055    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.764038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.764006    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65cr\" (UniqueName: \"kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr\") pod \"maas-api-revoke-keys-e2e-migrate-acace923-fxwmb\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:14.831558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.831523    4877 generic.go:356] "Generic (PLEG): container finished" podID="522a99c5-2000-4528-a092-bb00174b73f6" containerID="4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08" exitCode=0
Oct 30 03:12:14.831842 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.831575    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" event={"ID":"522a99c5-2000-4528-a092-bb00174b73f6","Type":"ContainerDied","Data":"4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"}
Oct 30 03:12:14.832259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.832240    4877 scope.go:122] "RemoveContainer" containerID="4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"
Oct 30 03:12:14.833810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.833774    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:14.833904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.833869    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:14.833943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.833915    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:14.867860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:14.867825    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:15.409447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:15.354856    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"]
Oct 30 03:12:15.847252 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:15.845781    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb" event={"ID":"77caf27c-c298-4d73-b935-c6e2e4ddd900","Type":"ContainerStarted","Data":"9e71c3996f08321295983e8e8a4d84ac81390cce76baa15ec0f88558213d9b3d"}
Oct 30 03:12:15.852182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:15.851233    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" event={"ID":"522a99c5-2000-4528-a092-bb00174b73f6","Type":"ContainerStarted","Data":"9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876"}
Oct 30 03:12:16.153666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:16.153585    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:12:16.153801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:16.153777    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:12:16.153846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:16.153826    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 03:12:16.257508 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:16.257469    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:12:16.753504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:16.753477    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:17.153735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:17.153696    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:12:17.153920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:17.153757    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 03:12:17.761850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:17.761824    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:18.157630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:18.157592    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:12:18.157793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:18.157664    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 03:12:18.770671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:18.769623    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:19.153640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:19.153565    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:12:19.153640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:19.153622    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 03:12:20.156938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:20.156895    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:12:20.157601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:20.156970    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 03:12:21.154259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:21.154219    4877 patch_prober.go:29] interesting pod/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:12:21.154530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:21.154502    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 03:12:21.949232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:21.949160    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb" event={"ID":"77caf27c-c298-4d73-b935-c6e2e4ddd900","Type":"ContainerStarted","Data":"e56e6469b0134e02be0858b90be3ebcdf5b25a625ac6269a88db714f8ac464d4"}
Oct 30 03:12:21.995825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:21.995761    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb" podStartSLOduration=1.952077377 podStartE2EDuration="7.995743159s" totalImagesPullingTime="6.043665782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:21.992799507 +0000 UTC m=+570.421688965" watchObservedRunningTime="2026-10-30 03:12:21.995743159 +0000 UTC m=+570.424632594"
Oct 30 03:12:22.159299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.159212    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:12:22.160494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.160469    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:12:22.161007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.160982    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:12:22.820160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.820129    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:22.958215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.958171    4877 generic.go:356] "Generic (PLEG): container finished" podID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerID="1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485" exitCode=0
Oct 30 03:12:22.958721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.958243    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" event={"ID":"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d","Type":"ContainerDied","Data":"1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"}
Oct 30 03:12:22.958783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.958754    4877 scope.go:122] "RemoveContainer" containerID="1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"
Oct 30 03:12:22.960232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.960208    4877 generic.go:356] "Generic (PLEG): container finished" podID="77caf27c-c298-4d73-b935-c6e2e4ddd900" containerID="e56e6469b0134e02be0858b90be3ebcdf5b25a625ac6269a88db714f8ac464d4" exitCode=0
Oct 30 03:12:22.960347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.960234    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:22.960347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.960306    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:22.960347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.960323    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb" event={"ID":"77caf27c-c298-4d73-b935-c6e2e4ddd900","Type":"ContainerDied","Data":"e56e6469b0134e02be0858b90be3ebcdf5b25a625ac6269a88db714f8ac464d4"}
Oct 30 03:12:22.960551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:22.960355    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:23.833448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:23.829276    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:23.972816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:23.972788    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" event={"ID":"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d","Type":"ContainerStarted","Data":"6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01"}
Oct 30 03:12:24.342630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.342601    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:24.420501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.403965    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:24.420501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.404066    4877 patch_prober.go:29] interesting pod/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:12:24.420501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.404129    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 03:12:24.458829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.454355    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr\" (UniqueName: \"kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr\") pod \"77caf27c-c298-4d73-b935-c6e2e4ddd900\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") "
Oct 30 03:12:24.458829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.454486    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca\") pod \"77caf27c-c298-4d73-b935-c6e2e4ddd900\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") "
Oct 30 03:12:24.458829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.454530    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca\") pod \"77caf27c-c298-4d73-b935-c6e2e4ddd900\" (UID: \"77caf27c-c298-4d73-b935-c6e2e4ddd900\") "
Oct 30 03:12:24.458829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.455973    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca" pod "77caf27c-c298-4d73-b935-c6e2e4ddd900" (UID: "77caf27c-c298-4d73-b935-c6e2e4ddd900"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:24.459217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.459122    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca" pod "77caf27c-c298-4d73-b935-c6e2e4ddd900" (UID: "77caf27c-c298-4d73-b935-c6e2e4ddd900"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:24.462774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.462750    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr" pod "77caf27c-c298-4d73-b935-c6e2e4ddd900" (UID: "77caf27c-c298-4d73-b935-c6e2e4ddd900"). InnerVolumeSpecName "kube-api-access-m65cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:24.556549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.556510    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65cr\" (UniqueName: \"kubernetes.io/projected/77caf27c-c298-4d73-b935-c6e2e4ddd900-kube-api-access-m65cr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.556549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.556547    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:24.556753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:24.556562    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77caf27c-c298-4d73-b935-c6e2e4ddd900-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:25.025518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.019777    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"
Oct 30 03:12:25.025518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.021493    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb" event={"ID":"77caf27c-c298-4d73-b935-c6e2e4ddd900","Type":"ContainerDied","Data":"9e71c3996f08321295983e8e8a4d84ac81390cce76baa15ec0f88558213d9b3d"}
Oct 30 03:12:25.025518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.021541    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e71c3996f08321295983e8e8a4d84ac81390cce76baa15ec0f88558213d9b3d"
Oct 30 03:12:25.380505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.377759    4877 patch_prober.go:29] interesting pod/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:12:25.380505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.377820    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 03:12:25.544340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.544288    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"]
Oct 30 03:12:25.545195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.545162    4877 cpu_manager.go:414] "containerMap: removing container" podUID="77caf27c-c298-4d73-b935-c6e2e4ddd900" containerName="revoke-keys"
Oct 30 03:12:25.545195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.545187    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77caf27c-c298-4d73-b935-c6e2e4ddd900" containerName="revoke-keys"
Oct 30 03:12:25.545392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.545355    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="77caf27c-c298-4d73-b935-c6e2e4ddd900" containerName="revoke-keys"
Oct 30 03:12:25.554212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.554193    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.561250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.561191    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"]
Oct 30 03:12:25.571309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.570880    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:12:25.584672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.584620    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:12:25.669645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.668986    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmvh\" (UniqueName: \"kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.669645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.669089    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.669645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.669110    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.769650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.769612    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmvh\" (UniqueName: \"kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.769840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.769799    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.769840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.769833    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.770626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.770603    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.773962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.773936    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.837446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.837200    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmvh\" (UniqueName: \"kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh\") pod \"maas-api-revoke-keys-e2e-ait-a788fe69-llf4j\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:25.872331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:25.872297    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:26.321916 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:26.321834    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b3dfe4_1a8f_45af_a59c_1622149f39a2.slice/crio-7c4cc191bd1729702541f79adae5c00889d68b9fae9a30c2691b0fffb493cd83 WatchSource:0}: Error finding container 7c4cc191bd1729702541f79adae5c00889d68b9fae9a30c2691b0fffb493cd83: Status 404 returned error can't find the container with id 7c4cc191bd1729702541f79adae5c00889d68b9fae9a30c2691b0fffb493cd83
Oct 30 03:12:26.323567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:26.323534    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"]
Oct 30 03:12:26.378671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:26.377915    4877 patch_prober.go:29] interesting pod/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:12:26.378671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:26.377984    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 03:12:27.058617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:27.058580    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j" event={"ID":"f2b3dfe4-1a8f-45af-a59c-1622149f39a2","Type":"ContainerStarted","Data":"23859c9eed461f089dbb783cd1a9634f8eb31758afe1325920f184f004633a84"}
Oct 30 03:12:27.060716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:27.058641    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j" event={"ID":"f2b3dfe4-1a8f-45af-a59c-1622149f39a2","Type":"ContainerStarted","Data":"7c4cc191bd1729702541f79adae5c00889d68b9fae9a30c2691b0fffb493cd83"}
Oct 30 03:12:27.133078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:27.133013    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j" podStartSLOduration=2.132994941 podStartE2EDuration="2.132994941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:27.132834536 +0000 UTC m=+575.561723974" watchObservedRunningTime="2026-10-30 03:12:27.132994941 +0000 UTC m=+575.561884362"
Oct 30 03:12:27.383044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:27.382771    4877 patch_prober.go:29] interesting pod/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:12:27.383044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:27.382832    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 03:12:27.717286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:27.717206    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"]
Oct 30 03:12:28.066355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.066314    4877 generic.go:356] "Generic (PLEG): container finished" podID="f2b3dfe4-1a8f-45af-a59c-1622149f39a2" containerID="23859c9eed461f089dbb783cd1a9634f8eb31758afe1325920f184f004633a84" exitCode=0
Oct 30 03:12:28.066355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.066350    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j" event={"ID":"f2b3dfe4-1a8f-45af-a59c-1622149f39a2","Type":"ContainerDied","Data":"23859c9eed461f089dbb783cd1a9634f8eb31758afe1325920f184f004633a84"}
Oct 30 03:12:28.068161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.068137    4877 generic.go:356] "Generic (PLEG): container finished" podID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerID="12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467" exitCode=0
Oct 30 03:12:28.068280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.068170    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" event={"ID":"733630f7-7e30-4437-bcbe-b13ad98034e4","Type":"ContainerDied","Data":"12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"}
Oct 30 03:12:28.068574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.068558    4877 scope.go:122] "RemoveContainer" containerID="12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"
Oct 30 03:12:28.069977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.069940    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:28.070076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.070025    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:28.070138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.070074    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:28.387500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.379776    4877 patch_prober.go:29] interesting pod/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:12:28.387500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:28.379834    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 03:12:29.093454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.093296    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" event={"ID":"733630f7-7e30-4437-bcbe-b13ad98034e4","Type":"ContainerStarted","Data":"b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242"}
Oct 30 03:12:29.352935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.352870    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:29.475060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.475019    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca\") pod \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") "
Oct 30 03:12:29.475568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.475093    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh\" (UniqueName: \"kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh\") pod \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") "
Oct 30 03:12:29.475568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.475171    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca\") pod \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\" (UID: \"f2b3dfe4-1a8f-45af-a59c-1622149f39a2\") "
Oct 30 03:12:29.475876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.475850    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca" pod "f2b3dfe4-1a8f-45af-a59c-1622149f39a2" (UID: "f2b3dfe4-1a8f-45af-a59c-1622149f39a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:29.485562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.482529    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca" pod "f2b3dfe4-1a8f-45af-a59c-1622149f39a2" (UID: "f2b3dfe4-1a8f-45af-a59c-1622149f39a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:29.485745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.485588    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh" pod "f2b3dfe4-1a8f-45af-a59c-1622149f39a2" (UID: "f2b3dfe4-1a8f-45af-a59c-1622149f39a2"). InnerVolumeSpecName "kube-api-access-gkmvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:29.588449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.576768    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:29.588449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.576813    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmvh\" (UniqueName: \"kubernetes.io/projected/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-kube-api-access-gkmvh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:29.588449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.576829    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b3dfe4-1a8f-45af-a59c-1622149f39a2-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:29.631275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.631078    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:29.631812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.631794    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:29.634345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.634326    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:29.718613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.718573    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:12:29.761678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.761611    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:12:29.761955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.761933    4877 patch_prober.go:29] interesting pod/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 03:12:29.762056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:29.761980    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 03:12:30.113448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:30.112576    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"
Oct 30 03:12:30.113448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:30.112943    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j" event={"ID":"f2b3dfe4-1a8f-45af-a59c-1622149f39a2","Type":"ContainerDied","Data":"7c4cc191bd1729702541f79adae5c00889d68b9fae9a30c2691b0fffb493cd83"}
Oct 30 03:12:30.113448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:30.112994    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4cc191bd1729702541f79adae5c00889d68b9fae9a30c2691b0fffb493cd83"
Oct 30 03:12:30.759980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:30.759945    4877 patch_prober.go:29] interesting pod/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 03:12:30.760373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:30.760000    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 03:12:31.120353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.120322    4877 generic.go:356] "Generic (PLEG): container finished" podID="f86a7659-10d5-4066-b558-c4db7c662362" containerID="904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a" exitCode=0
Oct 30 03:12:31.120584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.120393    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" event={"ID":"f86a7659-10d5-4066-b558-c4db7c662362","Type":"ContainerDied","Data":"904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"}
Oct 30 03:12:31.121006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.120987    4877 scope.go:122] "RemoveContainer" containerID="904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"
Oct 30 03:12:31.122161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.122132    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:31.122216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.122200    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:31.122262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.122232    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:31.759961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.759925    4877 patch_prober.go:29] interesting pod/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 03:12:31.760138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:31.759979    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 03:12:32.145917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.145873    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" event={"ID":"f86a7659-10d5-4066-b558-c4db7c662362","Type":"ContainerStarted","Data":"976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d"}
Oct 30 03:12:32.220445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.219809    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:32.227443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.224439    4877 patch_prober.go:29] interesting pod/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 03:12:32.227443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.224501    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 03:12:32.541022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.540982    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"]
Oct 30 03:12:32.541981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.541952    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f2b3dfe4-1a8f-45af-a59c-1622149f39a2" containerName="revoke-keys"
Oct 30 03:12:32.542138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.542124    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b3dfe4-1a8f-45af-a59c-1622149f39a2" containerName="revoke-keys"
Oct 30 03:12:32.542411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.542396    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b3dfe4-1a8f-45af-a59c-1622149f39a2" containerName="revoke-keys"
Oct 30 03:12:32.546315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.546300    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.561323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.561298    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:12:32.561912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.561891    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:12:32.565700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.565673    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"]
Oct 30 03:12:32.642295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.642259    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.642295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.642298    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.642516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.642363    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmrk\" (UniqueName: \"kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.721447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.720484    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h" podUID="7902e666-5bea-4428-bd6a-76399e41792c" containerName="payload-processing" containerID="cri-o://720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c" gracePeriod=25
Oct 30 03:12:32.743200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.743166    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.743862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.743841    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.744172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.744155    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmrk\" (UniqueName: \"kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.744655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.744629    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.745703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.745683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.759874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.759849    4877 patch_prober.go:29] interesting pod/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 03:12:32.759945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.759903    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 03:12:32.764142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.764125    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmrk\" (UniqueName: \"kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk\") pod \"maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:32.859132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:32.859096    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:33.156470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.151603    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:33.156470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.154104    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj\" (UniqueName: \"kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj\") pod \"7902e666-5bea-4428-bd6a-76399e41792c\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") "
Oct 30 03:12:33.156470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.154234    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume\") pod \"7902e666-5bea-4428-bd6a-76399e41792c\" (UID: \"7902e666-5bea-4428-bd6a-76399e41792c\") "
Oct 30 03:12:33.157686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.157640    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume" pod "7902e666-5bea-4428-bd6a-76399e41792c" (UID: "7902e666-5bea-4428-bd6a-76399e41792c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:33.160906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.160873    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj" pod "7902e666-5bea-4428-bd6a-76399e41792c" (UID: "7902e666-5bea-4428-bd6a-76399e41792c"). InnerVolumeSpecName "kube-api-access-2mcrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:33.163679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.163647    4877 generic.go:356] "Generic (PLEG): container finished" podID="7902e666-5bea-4428-bd6a-76399e41792c" containerID="720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c" exitCode=0
Oct 30 03:12:33.164579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.164560    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"
Oct 30 03:12:33.167167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.165828    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h" event={"ID":"7902e666-5bea-4428-bd6a-76399e41792c","Type":"ContainerDied","Data":"720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c"}
Oct 30 03:12:33.167167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.165884    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h" event={"ID":"7902e666-5bea-4428-bd6a-76399e41792c","Type":"ContainerDied","Data":"c6550f55e4c9c335aa7b3babfa2f8c123afa2140c50ef496e01f24b193a3e3b2"}
Oct 30 03:12:33.167167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.165915    4877 scope.go:122] "RemoveContainer" containerID="720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c"
Oct 30 03:12:33.207889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.206610    4877 scope.go:122] "RemoveContainer" containerID="720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c"
Oct 30 03:12:33.215487 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:33.208167    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c\": container with ID starting with 720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c not found: ID does not exist" containerID="720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c"
Oct 30 03:12:33.215487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.208205    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c"} err="failed to get container status \"720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c\": rpc error: code = NotFound desc = could not find container \"720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c\": container with ID starting with 720ccf8d907138556ea844cc7e1220c3075c3b95d478a7fa13d94d3a7a61806c not found: ID does not exist"
Oct 30 03:12:33.221207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.220777    4877 patch_prober.go:29] interesting pod/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 03:12:33.221207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.220831    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 03:12:33.252606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.252560    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"]
Oct 30 03:12:33.255364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.255308    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-421ef828-8568565666-qmh9h"]
Oct 30 03:12:33.258378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.258010    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mcrj\" (UniqueName: \"kubernetes.io/projected/7902e666-5bea-4428-bd6a-76399e41792c-kube-api-access-2mcrj\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:33.258378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.258032    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7902e666-5bea-4428-bd6a-76399e41792c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:33.258882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.258863    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"]
Oct 30 03:12:33.259676 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:33.259633    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644e54d5_08e4_4c2c_ae4f_1f88180202f2.slice/crio-6e5879753829230225b2a9544820cf5b645c74099423f856934413521f2f642c WatchSource:0}: Error finding container 6e5879753829230225b2a9544820cf5b645c74099423f856934413521f2f642c: Status 404 returned error can't find the container with id 6e5879753829230225b2a9544820cf5b645c74099423f856934413521f2f642c
Oct 30 03:12:33.729288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.729253    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7902e666-5bea-4428-bd6a-76399e41792c" path="/var/lib/kubelet/pods/7902e666-5bea-4428-bd6a-76399e41792c/volumes"
Oct 30 03:12:33.760165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.760140    4877 patch_prober.go:29] interesting pod/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 03:12:33.760263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:33.760181    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 03:12:34.172640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.172599    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8" event={"ID":"644e54d5-08e4-4c2c-ae4f-1f88180202f2","Type":"ContainerStarted","Data":"87b26226e005de82c805d9cfbeddab226aa3aaba078ecfe84c908a97bcde8ab0"}
Oct 30 03:12:34.173032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.172659    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8" event={"ID":"644e54d5-08e4-4c2c-ae4f-1f88180202f2","Type":"ContainerStarted","Data":"6e5879753829230225b2a9544820cf5b645c74099423f856934413521f2f642c"}
Oct 30 03:12:34.220062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.220022    4877 patch_prober.go:29] interesting pod/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 03:12:34.220220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.220081    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 03:12:34.230923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.230870    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8" podStartSLOduration=2.23085695 podStartE2EDuration="2.23085695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:34.229130412 +0000 UTC m=+582.658019882" watchObservedRunningTime="2026-10-30 03:12:34.23085695 +0000 UTC m=+582.659746368"
Oct 30 03:12:34.763963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.763886    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:12:34.764907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:34.764885    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:12:35.052906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:35.052872    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"]
Oct 30 03:12:35.205078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:35.205037    4877 generic.go:356] "Generic (PLEG): container finished" podID="644e54d5-08e4-4c2c-ae4f-1f88180202f2" containerID="87b26226e005de82c805d9cfbeddab226aa3aaba078ecfe84c908a97bcde8ab0" exitCode=0
Oct 30 03:12:35.205508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:35.205149    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8" event={"ID":"644e54d5-08e4-4c2c-ae4f-1f88180202f2","Type":"ContainerDied","Data":"87b26226e005de82c805d9cfbeddab226aa3aaba078ecfe84c908a97bcde8ab0"}
Oct 30 03:12:35.219156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:35.219127    4877 patch_prober.go:29] interesting pod/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 03:12:35.219277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:35.219179    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 03:12:36.219893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.219857    4877 patch_prober.go:29] interesting pod/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 03:12:36.220250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.219923    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 03:12:36.365806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.365778    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:36.419229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.419199    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk\" (UniqueName: \"kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk\") pod \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") "
Oct 30 03:12:36.419405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.419239    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca\") pod \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") "
Oct 30 03:12:36.419405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.419281    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca\") pod \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\" (UID: \"644e54d5-08e4-4c2c-ae4f-1f88180202f2\") "
Oct 30 03:12:36.419696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.419666    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca" pod "644e54d5-08e4-4c2c-ae4f-1f88180202f2" (UID: "644e54d5-08e4-4c2c-ae4f-1f88180202f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:36.421231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.421209    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca" pod "644e54d5-08e4-4c2c-ae4f-1f88180202f2" (UID: "644e54d5-08e4-4c2c-ae4f-1f88180202f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:12:36.421343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.421294    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk" pod "644e54d5-08e4-4c2c-ae4f-1f88180202f2" (UID: "644e54d5-08e4-4c2c-ae4f-1f88180202f2"). InnerVolumeSpecName "kube-api-access-jsmrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:36.520370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.520278    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsmrk\" (UniqueName: \"kubernetes.io/projected/644e54d5-08e4-4c2c-ae4f-1f88180202f2-kube-api-access-jsmrk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:36.520370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.520310    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:36.520370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.520321    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e54d5-08e4-4c2c-ae4f-1f88180202f2-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:36.608634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.608561    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"]
Oct 30 03:12:36.750409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:36.750366    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:37.219044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:37.218998    4877 patch_prober.go:29] interesting pod/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 03:12:37.219234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:37.219054    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 03:12:37.219234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:37.219081    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8" event={"ID":"644e54d5-08e4-4c2c-ae4f-1f88180202f2","Type":"ContainerDied","Data":"6e5879753829230225b2a9544820cf5b645c74099423f856934413521f2f642c"}
Oct 30 03:12:37.219234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:37.219111    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"
Oct 30 03:12:37.219234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:37.219125    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5879753829230225b2a9544820cf5b645c74099423f856934413521f2f642c"
Oct 30 03:12:38.224869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:38.224842    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:38.226082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:38.226060    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:38.805268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:38.805231    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"]
Oct 30 03:12:38.805529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:38.805498    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy" containerID="cri-o://6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01" gracePeriod=30
Oct 30 03:12:40.054114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.054072    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26" podUID="921202ff-7245-4756-b6e6-7ef958491b6e" containerName="payload-pre-processing" containerID="cri-o://769fcc60699a9a88982567071f08815a9a33e541480fc3babb0e334b87d26a09" gracePeriod=25
Oct 30 03:12:40.100858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.100825    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"]
Oct 30 03:12:40.101413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.101379    4877 cpu_manager.go:414] "containerMap: removing container" podUID="644e54d5-08e4-4c2c-ae4f-1f88180202f2" containerName="revoke-keys"
Oct 30 03:12:40.101413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.101398    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644e54d5-08e4-4c2c-ae4f-1f88180202f2" containerName="revoke-keys"
Oct 30 03:12:40.101629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.101453    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7902e666-5bea-4428-bd6a-76399e41792c" containerName="payload-processing"
Oct 30 03:12:40.101629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.101460    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7902e666-5bea-4428-bd6a-76399e41792c" containerName="payload-processing"
Oct 30 03:12:40.101629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.101568    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7902e666-5bea-4428-bd6a-76399e41792c" containerName="payload-processing"
Oct 30 03:12:40.101629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.101582    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="644e54d5-08e4-4c2c-ae4f-1f88180202f2" containerName="revoke-keys"
Oct 30 03:12:40.105173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.105153    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.122631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.122611    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a9c9b6-openshift-default-dockercfg-9jv62\""
Oct 30 03:12:40.143920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.143876    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"]
Oct 30 03:12:40.180128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180092    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180161    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180268    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180327    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djrkt\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180363    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180499    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180531    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180564    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.180864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.180624    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.247769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.247736    4877 generic.go:356] "Generic (PLEG): container finished" podID="921202ff-7245-4756-b6e6-7ef958491b6e" containerID="769fcc60699a9a88982567071f08815a9a33e541480fc3babb0e334b87d26a09" exitCode=0
Oct 30 03:12:40.247769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.247778    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26" event={"ID":"921202ff-7245-4756-b6e6-7ef958491b6e","Type":"ContainerDied","Data":"769fcc60699a9a88982567071f08815a9a33e541480fc3babb0e334b87d26a09"}
Oct 30 03:12:40.281293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281249    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281297    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281365    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281417    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281467    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281502    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djrkt\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281536    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281639    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.281826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.281674    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.282913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.282265    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.282913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.282492    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.282913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.282563    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.282913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.282731    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.283389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.283068    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.284542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.284502    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.285118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.285087    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.313377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.313356    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.313535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.313470    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djrkt\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.337929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.337902    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:40.382711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.382673    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume\") pod \"921202ff-7245-4756-b6e6-7ef958491b6e\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") "
Oct 30 03:12:40.382894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.382813    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6\" (UniqueName: \"kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6\") pod \"921202ff-7245-4756-b6e6-7ef958491b6e\" (UID: \"921202ff-7245-4756-b6e6-7ef958491b6e\") "
Oct 30 03:12:40.383031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.383008    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume" pod "921202ff-7245-4756-b6e6-7ef958491b6e" (UID: "921202ff-7245-4756-b6e6-7ef958491b6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:40.383193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.383175    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/921202ff-7245-4756-b6e6-7ef958491b6e-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:40.384996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.384970    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6" pod "921202ff-7245-4756-b6e6-7ef958491b6e" (UID: "921202ff-7245-4756-b6e6-7ef958491b6e"). InnerVolumeSpecName "kube-api-access-w8ht6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:40.446290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.446256    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:40.483961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.483923    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8ht6\" (UniqueName: \"kubernetes.io/projected/921202ff-7245-4756-b6e6-7ef958491b6e-kube-api-access-w8ht6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:40.783354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.783319    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"]
Oct 30 03:12:40.783577 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:40.783473    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713b5c0c_6af2_4b8b_a518_22df5d689146.slice/crio-2c5605da665cd65b28612e3d47b4cebff399039eff506f4d676f43276297fffb WatchSource:0}: Error finding container 2c5605da665cd65b28612e3d47b4cebff399039eff506f4d676f43276297fffb: Status 404 returned error can't find the container with id 2c5605da665cd65b28612e3d47b4cebff399039eff506f4d676f43276297fffb
Oct 30 03:12:40.786493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.786460    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:40.786574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.786527    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:40.786616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:40.786579    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:41.035792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.035706    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"]
Oct 30 03:12:41.038315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.038287    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a788fe69-llf4j"]
Oct 30 03:12:41.176765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.176729    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"]
Oct 30 03:12:41.177549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.177514    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy" containerID="cri-o://b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242" gracePeriod=30
Oct 30 03:12:41.254614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.254587    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"
Oct 30 03:12:41.254804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.254620    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26" event={"ID":"921202ff-7245-4756-b6e6-7ef958491b6e","Type":"ContainerDied","Data":"af328aad52b285afd0a239925c75ac8399cd1e4f32cc87e1bb1a42c6eda4651e"}
Oct 30 03:12:41.254804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.254691    4877 scope.go:122] "RemoveContainer" containerID="769fcc60699a9a88982567071f08815a9a33e541480fc3babb0e334b87d26a09"
Oct 30 03:12:41.260907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.260878    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" event={"ID":"713b5c0c-6af2-4b8b-a518-22df5d689146","Type":"ContainerStarted","Data":"fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"}
Oct 30 03:12:41.261034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.260924    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" event={"ID":"713b5c0c-6af2-4b8b-a518-22df5d689146","Type":"ContainerStarted","Data":"2c5605da665cd65b28612e3d47b4cebff399039eff506f4d676f43276297fffb"}
Oct 30 03:12:41.327667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.327543    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podStartSLOduration=1.327525476 podStartE2EDuration="1.327525476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:41.326989089 +0000 UTC m=+589.755878520" watchObservedRunningTime="2026-10-30 03:12:41.327525476 +0000 UTC m=+589.756414895"
Oct 30 03:12:41.369465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.369411    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"]
Oct 30 03:12:41.372461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.372432    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-acace923-85d999b499-wpb26"]
Oct 30 03:12:41.447004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.446966    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:41.452969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.452941    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:41.487055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.487023    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"]
Oct 30 03:12:41.488763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.488733    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-acace923-fxwmb"]
Oct 30 03:12:41.609671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.609536    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9" podUID="09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" containerName="payload-processing" containerID="cri-o://7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53" gracePeriod=25
Oct 30 03:12:41.637699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.637660    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"]
Oct 30 03:12:41.637988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.637938    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy" containerID="cri-o://9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876" gracePeriod=30
Oct 30 03:12:41.697256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.697225    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77caf27c-c298-4d73-b935-c6e2e4ddd900" path="/var/lib/kubelet/pods/77caf27c-c298-4d73-b935-c6e2e4ddd900/volumes"
Oct 30 03:12:41.697637 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.697622    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921202ff-7245-4756-b6e6-7ef958491b6e" path="/var/lib/kubelet/pods/921202ff-7245-4756-b6e6-7ef958491b6e/volumes"
Oct 30 03:12:41.697986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.697974    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b3dfe4-1a8f-45af-a59c-1622149f39a2" path="/var/lib/kubelet/pods/f2b3dfe4-1a8f-45af-a59c-1622149f39a2/volumes"
Oct 30 03:12:41.897380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:41.897349    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:42.009089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.009058    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x\" (UniqueName: \"kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x\") pod \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") "
Oct 30 03:12:42.009265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.009111    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume\") pod \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\" (UID: \"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f\") "
Oct 30 03:12:42.009511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.009487    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume" pod "09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" (UID: "09fd0ccf-4e7a-4e88-ba6b-4d673711f48f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:42.011301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.011275    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x" pod "09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" (UID: "09fd0ccf-4e7a-4e88-ba6b-4d673711f48f"). InnerVolumeSpecName "kube-api-access-8tq8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:42.111014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.110974    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tq8x\" (UniqueName: \"kubernetes.io/projected/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-kube-api-access-8tq8x\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:42.111014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.111011    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:42.271791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.271759    4877 generic.go:356] "Generic (PLEG): container finished" podID="09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" containerID="7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53" exitCode=0
Oct 30 03:12:42.272274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.271817    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"
Oct 30 03:12:42.272274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.271850    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9" event={"ID":"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f","Type":"ContainerDied","Data":"7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53"}
Oct 30 03:12:42.272274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.271920    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9" event={"ID":"09fd0ccf-4e7a-4e88-ba6b-4d673711f48f","Type":"ContainerDied","Data":"84a7a25ed53af5a0914e89b36e56047a2d40b45f1876ed684770e78c20b6f2a4"}
Oct 30 03:12:42.272274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.271957    4877 scope.go:122] "RemoveContainer" containerID="7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53"
Oct 30 03:12:42.272956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.272255    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:42.273729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.273710    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:12:42.305245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.304315    4877 scope.go:122] "RemoveContainer" containerID="7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53"
Oct 30 03:12:42.313317 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:42.312346    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53\": container with ID starting with 7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53 not found: ID does not exist" containerID="7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53"
Oct 30 03:12:42.313317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.312394    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53"} err="failed to get container status \"7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53\": rpc error: code = NotFound desc = could not find container \"7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53\": container with ID starting with 7de684d07f829958a4897d9e82fb4a544ecd4abe7f6b2ccd32070af145645c53 not found: ID does not exist"
Oct 30 03:12:42.388002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.387894    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"]
Oct 30 03:12:42.401774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:42.401743    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c7c56c3d-8956cc6f4-twwb9"]
Oct 30 03:12:43.016354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:43.016319    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"]
Oct 30 03:12:43.067400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:43.067366    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"]
Oct 30 03:12:43.697378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:43.697344    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" path="/var/lib/kubelet/pods/09fd0ccf-4e7a-4e88-ba6b-4d673711f48f/volumes"
Oct 30 03:12:44.056126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.056102    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:44.132276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132247    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132284    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132330    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132355    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132385    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132482    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132560    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132585    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:44.132680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132594    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132681    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert\") pod \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\" (UID: \"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d\") "
Oct 30 03:12:44.132894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132703    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:44.132894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.132765    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:44.133159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.133106    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.133159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.133129    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.133159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.133145    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.133345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.133197    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:44.133483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.133460    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:44.135197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.135170    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "kube-api-access-5jdpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:44.135268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.135240    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:44.135393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.135369    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:44.136519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.136497    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo" pod "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" (UID: "f6fa12d0-de53-4f30-ba56-dd3f8ea1383d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:44.234602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.234501    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdpk\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-kube-api-access-5jdpk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.234602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.234546    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.234602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.234561    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.234602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.234572    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.234602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.234583    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.234602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.234595    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:44.287647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.287607    4877 generic.go:356] "Generic (PLEG): container finished" podID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerID="6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01" exitCode=0
Oct 30 03:12:44.287827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.287681    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"
Oct 30 03:12:44.287827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.287697    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" event={"ID":"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d","Type":"ContainerDied","Data":"6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01"}
Oct 30 03:12:44.287827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.287792    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2" event={"ID":"f6fa12d0-de53-4f30-ba56-dd3f8ea1383d","Type":"ContainerDied","Data":"152f94fefb3734c912e695e27e769ca83dbb5cc7b7bf8b298412a86d2b1bda5a"}
Oct 30 03:12:44.287827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.287823    4877 scope.go:122] "RemoveContainer" containerID="6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01"
Oct 30 03:12:44.297784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.297766    4877 scope.go:122] "RemoveContainer" containerID="1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"
Oct 30 03:12:44.306630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.306595    4877 scope.go:122] "RemoveContainer" containerID="6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01"
Oct 30 03:12:44.306959 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:44.306918    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01\": container with ID starting with 6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01 not found: ID does not exist" containerID="6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01"
Oct 30 03:12:44.306959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.306949    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01"} err="failed to get container status \"6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01\": rpc error: code = NotFound desc = could not find container \"6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01\": container with ID starting with 6116d1cd5cf32ffa79abd458f9035965456fc99a4c643450be20d2d6385eee01 not found: ID does not exist"
Oct 30 03:12:44.307136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.306967    4877 scope.go:122] "RemoveContainer" containerID="1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"
Oct 30 03:12:44.307258 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:44.307240    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485\": container with ID starting with 1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485 not found: ID does not exist" containerID="1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"
Oct 30 03:12:44.307294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.307262    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485"} err="failed to get container status \"1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485\": rpc error: code = NotFound desc = could not find container \"1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485\": container with ID starting with 1b158187bfa6bce86c7b5a1d823fe0acd25a35caad50986688fbd576a8b80485 not found: ID does not exist"
Oct 30 03:12:44.339883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.339856    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"]
Oct 30 03:12:44.343401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.343381    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-421ef828-openshift-default-59759b6669-cnhw2"]
Oct 30 03:12:44.812579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.808107    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"]
Oct 30 03:12:44.813238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.812765    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3f9206f0-9qql8"]
Oct 30 03:12:44.931225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.931192    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"]
Oct 30 03:12:44.931509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:44.931479    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy" containerID="cri-o://976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d" gracePeriod=30
Oct 30 03:12:45.697343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.697305    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644e54d5-08e4-4c2c-ae4f-1f88180202f2" path="/var/lib/kubelet/pods/644e54d5-08e4-4c2c-ae4f-1f88180202f2/volumes"
Oct 30 03:12:45.697885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.697856    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" path="/var/lib/kubelet/pods/f6fa12d0-de53-4f30-ba56-dd3f8ea1383d/volumes"
Oct 30 03:12:45.857196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857159    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"]
Oct 30 03:12:45.857649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857636    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy"
Oct 30 03:12:45.857695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857650    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy"
Oct 30 03:12:45.857695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857663    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy"
Oct 30 03:12:45.857695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857668    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy"
Oct 30 03:12:45.857695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857679    4877 cpu_manager.go:414] "containerMap: removing container" podUID="09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" containerName="payload-processing"
Oct 30 03:12:45.857695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857685    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" containerName="payload-processing"
Oct 30 03:12:45.857836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857706    4877 cpu_manager.go:414] "containerMap: removing container" podUID="921202ff-7245-4756-b6e6-7ef958491b6e" containerName="payload-pre-processing"
Oct 30 03:12:45.857836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857711    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921202ff-7245-4756-b6e6-7ef958491b6e" containerName="payload-pre-processing"
Oct 30 03:12:45.857836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857779    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy"
Oct 30 03:12:45.857836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857787    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fa12d0-de53-4f30-ba56-dd3f8ea1383d" containerName="istio-proxy"
Oct 30 03:12:45.857836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857794    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fd0ccf-4e7a-4e88-ba6b-4d673711f48f" containerName="payload-processing"
Oct 30 03:12:45.857836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.857803    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="921202ff-7245-4756-b6e6-7ef958491b6e" containerName="payload-pre-processing"
Oct 30 03:12:45.861200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.861179    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:45.882350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.882322    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"]
Oct 30 03:12:45.890496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.890473    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a9c9b6\""
Oct 30 03:12:45.890602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.890526    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a9c9b6-dockercfg-2rcsm\""
Oct 30 03:12:45.927638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.927612    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"]
Oct 30 03:12:45.931275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.931258    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:45.962497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.954153    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"]
Oct 30 03:12:45.962497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.955080    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:45.962497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:45.955227    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrvr\" (UniqueName: \"kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr\") pod \"payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:46.056513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.056481    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.056693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.056540    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrvr\" (UniqueName: \"kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr\") pod \"payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:46.056693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.056672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:46.056793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.056729    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkfp\" (UniqueName: \"kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp\") pod \"payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.057478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.057412    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:46.081717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.081683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrvr\" (UniqueName: \"kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr\") pod \"payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:46.157446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.157387    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkfp\" (UniqueName: \"kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp\") pod \"payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.157613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.157502    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.158184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.158165    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.174689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.174654    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:46.176690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.176668    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkfp\" (UniqueName: \"kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp\") pod \"payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.235874 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:46.235677    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:12:46.251711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.251034    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:46.258822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.257755    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"]
Oct 30 03:12:46.263571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.263548    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.279876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.279828    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e44313-openshift-default-dockercfg-tlmnm\""
Oct 30 03:12:46.292528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.292499    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"]
Oct 30 03:12:46.360206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360169    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360535    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360607    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360637    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360662    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360714    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360757    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.360848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360791    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.361071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.360927    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztlkb\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468084    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlkb\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468180    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468206    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468237    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468264    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468289    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468340    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468376    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.470528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.468410    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.474546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.472504    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.474546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.472769    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.474546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.473016    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.474546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.473251    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.474546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.473914    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.474883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.474643    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.479078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.478979    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.491824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.491796    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlkb\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.492714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.492691    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.511282 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.511251    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"]
Oct 30 03:12:46.512623 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:46.512583    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eae2566_d17c_4e70_96bc_c417fa401866.slice/crio-8060515aa20d5594ddba0ad293e345343b3abf4ef2dff9cd4416ab36718bfa5e WatchSource:0}: Error finding container 8060515aa20d5594ddba0ad293e345343b3abf4ef2dff9cd4416ab36718bfa5e: Status 404 returned error can't find the container with id 8060515aa20d5594ddba0ad293e345343b3abf4ef2dff9cd4416ab36718bfa5e
Oct 30 03:12:46.520607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.520586    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:12:46.582008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.581982    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"]
Oct 30 03:12:46.582596 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:46.582113    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62ea3ee_bba8_42d1_98b7_8bb07ee9c286.slice/crio-93047dd775fbf6b1757b363d0f9cffbf424b189c287643b3da6282fbcc875998 WatchSource:0}: Error finding container 93047dd775fbf6b1757b363d0f9cffbf424b189c287643b3da6282fbcc875998: Status 404 returned error can't find the container with id 93047dd775fbf6b1757b363d0f9cffbf424b189c287643b3da6282fbcc875998
Oct 30 03:12:46.596949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.596918    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.670750    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.670844    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.670884    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.670932    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.671028    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.671067    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.671092    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.671210    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.671336    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs\") pod \"733630f7-7e30-4437-bcbe-b13ad98034e4\" (UID: \"733630f7-7e30-4437-bcbe-b13ad98034e4\") "
Oct 30 03:12:46.672595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.672511    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.674323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.672853    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:46.674323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.673071    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.674323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.673235    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.674323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.674220    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.676153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.676115    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:46.686677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.686632    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "kube-api-access-ff2k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:46.686784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.686692    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.687729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.687698    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo" pod "733630f7-7e30-4437-bcbe-b13ad98034e4" (UID: "733630f7-7e30-4437-bcbe-b13ad98034e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:46.772982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.772904    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.772982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.772945    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773272    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733630f7-7e30-4437-bcbe-b13ad98034e4-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773300    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773314    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773330    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773345    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2k6\" (UniqueName: \"kubernetes.io/projected/733630f7-7e30-4437-bcbe-b13ad98034e4-kube-api-access-ff2k6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773359    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.773395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.773372    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733630f7-7e30-4437-bcbe-b13ad98034e4-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.887876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.887847    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:12:46.933608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.933577    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"]
Oct 30 03:12:46.934853 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:46.934806    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d09e0b5_a925_4499_973c_85092ecdf66b.slice/crio-b181e3805993215c6cff5af28717c4e1c0224527adad89100db33ac84921661d WatchSource:0}: Error finding container b181e3805993215c6cff5af28717c4e1c0224527adad89100db33ac84921661d: Status 404 returned error can't find the container with id b181e3805993215c6cff5af28717c4e1c0224527adad89100db33ac84921661d
Oct 30 03:12:46.937861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.937819    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:46.937957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.937902    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:46.937957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.937951    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:12:46.976099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976064    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976165    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976207    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976247    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976290    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976331    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.976450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976373    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976447    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976485    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976531    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket\") pod \"522a99c5-2000-4528-a092-bb00174b73f6\" (UID: \"522a99c5-2000-4528-a092-bb00174b73f6\") "
Oct 30 03:12:46.976755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976566    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.976755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976578    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.976940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976918    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.977074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976947    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.977074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976965    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:46.977074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.976917    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:46.977276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.977150    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.978680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.978645    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:46.979281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.978872    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "kube-api-access-v7j75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:46.979281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.979202    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:46.979442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:46.979275    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo" pod "522a99c5-2000-4528-a092-bb00174b73f6" (UID: "522a99c5-2000-4528-a092-bb00174b73f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:47.077535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.077501    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7j75\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-kube-api-access-v7j75\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:47.077535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.077531    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522a99c5-2000-4528-a092-bb00174b73f6-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:47.077535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.077541    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522a99c5-2000-4528-a092-bb00174b73f6-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:47.077828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.077549    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522a99c5-2000-4528-a092-bb00174b73f6-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:47.077828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.077558    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:47.077828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.077566    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522a99c5-2000-4528-a092-bb00174b73f6-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:47.315324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.315288    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" event={"ID":"2d09e0b5-a925-4499-973c-85092ecdf66b","Type":"ContainerStarted","Data":"fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"}
Oct 30 03:12:47.315536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.315352    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" event={"ID":"2d09e0b5-a925-4499-973c-85092ecdf66b","Type":"ContainerStarted","Data":"b181e3805993215c6cff5af28717c4e1c0224527adad89100db33ac84921661d"}
Oct 30 03:12:47.317187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.317161    4877 generic.go:356] "Generic (PLEG): container finished" podID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerID="b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242" exitCode=0
Oct 30 03:12:47.317315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.317218    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"
Oct 30 03:12:47.317315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.317241    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" event={"ID":"733630f7-7e30-4437-bcbe-b13ad98034e4","Type":"ContainerDied","Data":"b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242"}
Oct 30 03:12:47.317315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.317289    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w" event={"ID":"733630f7-7e30-4437-bcbe-b13ad98034e4","Type":"ContainerDied","Data":"168db24a95e33bbafbc3c0b3bb4403552c5b261588ef5f5b6095c19450ebce46"}
Oct 30 03:12:47.317535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.317318    4877 scope.go:122] "RemoveContainer" containerID="b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242"
Oct 30 03:12:47.319703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.319676    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" event={"ID":"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286","Type":"ContainerStarted","Data":"a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94"}
Oct 30 03:12:47.319926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.319725    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" event={"ID":"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286","Type":"ContainerStarted","Data":"93047dd775fbf6b1757b363d0f9cffbf424b189c287643b3da6282fbcc875998"}
Oct 30 03:12:47.319926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.319795    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:47.321829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.321527    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" event={"ID":"3eae2566-d17c-4e70-96bc-c417fa401866","Type":"ContainerStarted","Data":"cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418"}
Oct 30 03:12:47.321829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.321571    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" event={"ID":"3eae2566-d17c-4e70-96bc-c417fa401866","Type":"ContainerStarted","Data":"8060515aa20d5594ddba0ad293e345343b3abf4ef2dff9cd4416ab36718bfa5e"}
Oct 30 03:12:47.321829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.321645    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:12:47.323253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.323230    4877 generic.go:356] "Generic (PLEG): container finished" podID="522a99c5-2000-4528-a092-bb00174b73f6" containerID="9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876" exitCode=0
Oct 30 03:12:47.323362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.323269    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"
Oct 30 03:12:47.323438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.323372    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" event={"ID":"522a99c5-2000-4528-a092-bb00174b73f6","Type":"ContainerDied","Data":"9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876"}
Oct 30 03:12:47.323438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.323404    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz" event={"ID":"522a99c5-2000-4528-a092-bb00174b73f6","Type":"ContainerDied","Data":"e72572fb4a73e050119e7cc22c310589c4b64a6c8003cea74055835dc71aea1f"}
Oct 30 03:12:47.328876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.328856    4877 scope.go:122] "RemoveContainer" containerID="12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"
Oct 30 03:12:47.338280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.338259    4877 scope.go:122] "RemoveContainer" containerID="b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242"
Oct 30 03:12:47.338651 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:47.338626    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242\": container with ID starting with b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242 not found: ID does not exist" containerID="b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242"
Oct 30 03:12:47.338757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.338663    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242"} err="failed to get container status \"b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242\": rpc error: code = NotFound desc = could not find container \"b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242\": container with ID starting with b34bdfd72532edcd0c1e1d8022490f2d152e0a8e0b918d9b10953c06b00c4242 not found: ID does not exist"
Oct 30 03:12:47.338757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.338687    4877 scope.go:122] "RemoveContainer" containerID="12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"
Oct 30 03:12:47.338994 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:47.338969    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467\": container with ID starting with 12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467 not found: ID does not exist" containerID="12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"
Oct 30 03:12:47.339049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.339004    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467"} err="failed to get container status \"12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467\": rpc error: code = NotFound desc = could not find container \"12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467\": container with ID starting with 12df0f51bd6a5a3ab92601a5263225c3a0fe63ac8ed01e6c696b71fb9bacd467 not found: ID does not exist"
Oct 30 03:12:47.339049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.339026    4877 scope.go:122] "RemoveContainer" containerID="9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876"
Oct 30 03:12:47.348927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.348908    4877 scope.go:122] "RemoveContainer" containerID="4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"
Oct 30 03:12:47.358207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.358183    4877 scope.go:122] "RemoveContainer" containerID="9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876"
Oct 30 03:12:47.358626 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:47.358599    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876\": container with ID starting with 9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876 not found: ID does not exist" containerID="9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876"
Oct 30 03:12:47.358703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.358632    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876"} err="failed to get container status \"9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876\": rpc error: code = NotFound desc = could not find container \"9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876\": container with ID starting with 9d4a546c0061c30ec99dea86b194ba6693458d01cea2c794c8abf2e4fd640876 not found: ID does not exist"
Oct 30 03:12:47.358703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.358651    4877 scope.go:122] "RemoveContainer" containerID="4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"
Oct 30 03:12:47.358924 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:47.358909    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08\": container with ID starting with 4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08 not found: ID does not exist" containerID="4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"
Oct 30 03:12:47.358969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.358931    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08"} err="failed to get container status \"4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08\": rpc error: code = NotFound desc = could not find container \"4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08\": container with ID starting with 4443aed9f9690d8193d0b28fe2c73ff50b65cd48df885c36be787cc73fc18a08 not found: ID does not exist"
Oct 30 03:12:47.362801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.362755    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podStartSLOduration=1.3627429979999999 podStartE2EDuration="1.362742998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:47.360231883 +0000 UTC m=+595.789121302" watchObservedRunningTime="2026-10-30 03:12:47.362742998 +0000 UTC m=+595.791632417"
Oct 30 03:12:47.405040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.405004    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"]
Oct 30 03:12:47.408850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.408825    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-acace923-gw-openshift-default-77b6d8d9df-ljsnz"]
Oct 30 03:12:47.444644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.444615    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"]
Oct 30 03:12:47.455683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.455649    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a788fe69-openshift-default-86bb65fb5c-5t46w"]
Oct 30 03:12:47.495810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.495758    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" podStartSLOduration=2.495742471 podStartE2EDuration="2.495742471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:47.492918125 +0000 UTC m=+595.921807544" watchObservedRunningTime="2026-10-30 03:12:47.495742471 +0000 UTC m=+595.924631888"
Oct 30 03:12:47.533236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.533177    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" podStartSLOduration=2.533158936 podStartE2EDuration="2.533158936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:47.531663409 +0000 UTC m=+595.960552838" watchObservedRunningTime="2026-10-30 03:12:47.533158936 +0000 UTC m=+595.962048354"
Oct 30 03:12:47.598023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.597945    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:47.697629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.697593    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522a99c5-2000-4528-a092-bb00174b73f6" path="/var/lib/kubelet/pods/522a99c5-2000-4528-a092-bb00174b73f6/volumes"
Oct 30 03:12:47.698044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:47.698029    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" path="/var/lib/kubelet/pods/733630f7-7e30-4437-bcbe-b13ad98034e4/volumes"
Oct 30 03:12:48.017164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.017063    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl" podUID="34a58d2f-2944-4520-a3b0-e6ae65984f2b" containerName="payload-processing" containerID="cri-o://e7e521ebb13c57679a2d531434ac44afd0472f78cea20f5124e25087999b3d4c" gracePeriod=25
Oct 30 03:12:48.067910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.067869    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8" podUID="1514bc1c-9d56-48a2-864f-8640bf315dda" containerName="payload-pre-processing" containerID="cri-o://b9ffd31869a7677a7f370c66213059798d422a9829fd8f2b2746408eb21a815e" gracePeriod=25
Oct 30 03:12:48.258827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.258790    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"]
Oct 30 03:12:48.259693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259671    4877 cpu_manager.go:414] "containerMap: removing container" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy"
Oct 30 03:12:48.259753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259697    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy"
Oct 30 03:12:48.259753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259713    4877 cpu_manager.go:414] "containerMap: removing container" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy"
Oct 30 03:12:48.259753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259721    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy"
Oct 30 03:12:48.259753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259741    4877 cpu_manager.go:414] "containerMap: removing container" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy"
Oct 30 03:12:48.259753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259748    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy"
Oct 30 03:12:48.259908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259856    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy"
Oct 30 03:12:48.259908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259874    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy"
Oct 30 03:12:48.259908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.259885    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="522a99c5-2000-4528-a092-bb00174b73f6" containerName="istio-proxy"
Oct 30 03:12:48.263592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.263573    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.284528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.284351    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2d5b09\""
Oct 30 03:12:48.285571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.285546    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"]
Oct 30 03:12:48.319500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.319403    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"]
Oct 30 03:12:48.320333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.320305    4877 cpu_manager.go:414] "containerMap: removing container" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy"
Oct 30 03:12:48.320434 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.320336    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy"
Oct 30 03:12:48.320514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.320501    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="733630f7-7e30-4437-bcbe-b13ad98034e4" containerName="istio-proxy"
Oct 30 03:12:48.325683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.325391    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.340708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.340677    4877 generic.go:356] "Generic (PLEG): container finished" podID="34a58d2f-2944-4520-a3b0-e6ae65984f2b" containerID="e7e521ebb13c57679a2d531434ac44afd0472f78cea20f5124e25087999b3d4c" exitCode=0
Oct 30 03:12:48.340831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.340713    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl" event={"ID":"34a58d2f-2944-4520-a3b0-e6ae65984f2b","Type":"ContainerDied","Data":"e7e521ebb13c57679a2d531434ac44afd0472f78cea20f5124e25087999b3d4c"}
Oct 30 03:12:48.340831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.340774    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl" event={"ID":"34a58d2f-2944-4520-a3b0-e6ae65984f2b","Type":"ContainerDied","Data":"cef733607875d2929f6b3ac49ce4b4fd7b4f726cf0cbcf0b288e2493f315340d"}
Oct 30 03:12:48.340831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.340802    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef733607875d2929f6b3ac49ce4b4fd7b4f726cf0cbcf0b288e2493f315340d"
Oct 30 03:12:48.342910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.342882    4877 generic.go:356] "Generic (PLEG): container finished" podID="1514bc1c-9d56-48a2-864f-8640bf315dda" containerID="b9ffd31869a7677a7f370c66213059798d422a9829fd8f2b2746408eb21a815e" exitCode=0
Oct 30 03:12:48.343050 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.343029    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8" event={"ID":"1514bc1c-9d56-48a2-864f-8640bf315dda","Type":"ContainerDied","Data":"b9ffd31869a7677a7f370c66213059798d422a9829fd8f2b2746408eb21a815e"}
Oct 30 03:12:48.344569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.344504    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2d5b09-dockercfg-djh6z\""
Oct 30 03:12:48.346518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.346497    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:48.348773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.348750    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"]
Oct 30 03:12:48.392437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.392393    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.392645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.392626    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54plq\" (UniqueName: \"kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq\") pod \"payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.411679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.411655    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:48.493911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.493878    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q\" (UniqueName: \"kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q\") pod \"1514bc1c-9d56-48a2-864f-8640bf315dda\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") "
Oct 30 03:12:48.494582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.494558    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume\") pod \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") "
Oct 30 03:12:48.494783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.494766    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6\" (UniqueName: \"kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6\") pod \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\" (UID: \"34a58d2f-2944-4520-a3b0-e6ae65984f2b\") "
Oct 30 03:12:48.494934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.494918    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume\") pod \"1514bc1c-9d56-48a2-864f-8640bf315dda\" (UID: \"1514bc1c-9d56-48a2-864f-8640bf315dda\") "
Oct 30 03:12:48.505115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.505078    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54plq\" (UniqueName: \"kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq\") pod \"payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.505295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.505132    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9fq\" (UniqueName: \"kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq\") pod \"payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.505295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.505222    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.505480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.505302    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.506224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.506197    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.508672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.508644    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume" pod "1514bc1c-9d56-48a2-864f-8640bf315dda" (UID: "1514bc1c-9d56-48a2-864f-8640bf315dda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:48.509185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.509163    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume" pod "34a58d2f-2944-4520-a3b0-e6ae65984f2b" (UID: "34a58d2f-2944-4520-a3b0-e6ae65984f2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:48.529790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.529415    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q" pod "1514bc1c-9d56-48a2-864f-8640bf315dda" (UID: "1514bc1c-9d56-48a2-864f-8640bf315dda"). InnerVolumeSpecName "kube-api-access-nsf5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:48.532898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.532744    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54plq\" (UniqueName: \"kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq\") pod \"payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.537758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.537728    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6" pod "34a58d2f-2944-4520-a3b0-e6ae65984f2b" (UID: "34a58d2f-2944-4520-a3b0-e6ae65984f2b"). InnerVolumeSpecName "kube-api-access-hvrs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:48.580537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.580499    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:48.603773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.603749    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:48.606279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606254    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9fq\" (UniqueName: \"kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq\") pod \"payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.606382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606303    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.606382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606361    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsf5q\" (UniqueName: \"kubernetes.io/projected/1514bc1c-9d56-48a2-864f-8640bf315dda-kube-api-access-nsf5q\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:48.606382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606375    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a58d2f-2944-4520-a3b0-e6ae65984f2b-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:48.606500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606388    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrs6\" (UniqueName: \"kubernetes.io/projected/34a58d2f-2944-4520-a3b0-e6ae65984f2b-kube-api-access-hvrs6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:48.606500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606403    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1514bc1c-9d56-48a2-864f-8640bf315dda-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:48.606954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.606934    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.625972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.625932    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9fq\" (UniqueName: \"kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq\") pod \"payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.654167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.654067    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:48.892723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.892692    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"]
Oct 30 03:12:48.893145 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:48.893118    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e19aa13_f266_4f60_b227_5c7b2ce8031a.slice/crio-5fdadf837269f51d150fe28238e0fb7c7d306092f2cdbfed96f8de77dc799a50 WatchSource:0}: Error finding container 5fdadf837269f51d150fe28238e0fb7c7d306092f2cdbfed96f8de77dc799a50: Status 404 returned error can't find the container with id 5fdadf837269f51d150fe28238e0fb7c7d306092f2cdbfed96f8de77dc799a50
Oct 30 03:12:48.968705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:48.968676    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"]
Oct 30 03:12:48.969317 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:48.969288    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5448a7de_48a8_4a48_9674_c113cbf14868.slice/crio-8247f6f93da3773f16543bf9d7dea21e0f8a9a8ae80dfe40a103dd904c1d1782 WatchSource:0}: Error finding container 8247f6f93da3773f16543bf9d7dea21e0f8a9a8ae80dfe40a103dd904c1d1782: Status 404 returned error can't find the container with id 8247f6f93da3773f16543bf9d7dea21e0f8a9a8ae80dfe40a103dd904c1d1782
Oct 30 03:12:49.351803 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.351766    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" event={"ID":"5448a7de-48a8-4a48-9674-c113cbf14868","Type":"ContainerStarted","Data":"1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6"}
Oct 30 03:12:49.352294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.351857    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" event={"ID":"5448a7de-48a8-4a48-9674-c113cbf14868","Type":"ContainerStarted","Data":"8247f6f93da3773f16543bf9d7dea21e0f8a9a8ae80dfe40a103dd904c1d1782"}
Oct 30 03:12:49.352294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.351884    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:12:49.355211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.355180    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" event={"ID":"6e19aa13-f266-4f60-b227-5c7b2ce8031a","Type":"ContainerStarted","Data":"65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c"}
Oct 30 03:12:49.355371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.355234    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" event={"ID":"6e19aa13-f266-4f60-b227-5c7b2ce8031a","Type":"ContainerStarted","Data":"5fdadf837269f51d150fe28238e0fb7c7d306092f2cdbfed96f8de77dc799a50"}
Oct 30 03:12:49.355371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.355276    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:12:49.356786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.356765    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"
Oct 30 03:12:49.356891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.356765    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"
Oct 30 03:12:49.356891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.356762    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8" event={"ID":"1514bc1c-9d56-48a2-864f-8640bf315dda","Type":"ContainerDied","Data":"fe7a1f1cadcf6af28ace76a0c0802a436afc30c8569eaf8043be1738ff200424"}
Oct 30 03:12:49.357001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.356897    4877 scope.go:122] "RemoveContainer" containerID="b9ffd31869a7677a7f370c66213059798d422a9829fd8f2b2746408eb21a815e"
Oct 30 03:12:49.357256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.357238    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:49.359087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.359073    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:12:49.405074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.405021    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" podStartSLOduration=1.405006189 podStartE2EDuration="1.405006189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:49.389688574 +0000 UTC m=+597.818578011" watchObservedRunningTime="2026-10-30 03:12:49.405006189 +0000 UTC m=+597.833895640"
Oct 30 03:12:49.429406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.429358    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" podStartSLOduration=1.429345995 podStartE2EDuration="1.429345995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:49.426888229 +0000 UTC m=+597.855777674" watchObservedRunningTime="2026-10-30 03:12:49.429345995 +0000 UTC m=+597.858235413"
Oct 30 03:12:49.462083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.462051    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"]
Oct 30 03:12:49.469602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.469575    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3f9206f0-6f45b6df9f-lllv8"]
Oct 30 03:12:49.530394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.530357    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"]
Oct 30 03:12:49.542884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.542851    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3f9206f0-76cf568f68-pnmwl"]
Oct 30 03:12:49.700069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.699992    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1514bc1c-9d56-48a2-864f-8640bf315dda" path="/var/lib/kubelet/pods/1514bc1c-9d56-48a2-864f-8640bf315dda/volumes"
Oct 30 03:12:49.700522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:49.700506    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a58d2f-2944-4520-a3b0-e6ae65984f2b" path="/var/lib/kubelet/pods/34a58d2f-2944-4520-a3b0-e6ae65984f2b/volumes"
Oct 30 03:12:50.193474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.193448    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:50.325792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.325760    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.325812    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.325884    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.325925    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.325942    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.325990    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326033    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326064    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326097    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:50.326279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326122    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token\") pod \"f86a7659-10d5-4066-b558-c4db7c662362\" (UID: \"f86a7659-10d5-4066-b558-c4db7c662362\") "
Oct 30 03:12:50.326523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326301    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:50.326523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326324    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:50.326523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326376    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:50.326523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326404    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:12:50.326787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326669    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.326787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326691    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f86a7659-10d5-4066-b558-c4db7c662362-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.326787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326705    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.326787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326718    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.326787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.326732    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.328381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.328341    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:12:50.328527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.328357    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:12:50.328527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.328413    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:50.328638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.328523    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx" pod "f86a7659-10d5-4066-b558-c4db7c662362" (UID: "f86a7659-10d5-4066-b558-c4db7c662362"). InnerVolumeSpecName "kube-api-access-9fkxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:12:50.365651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.365618    4877 generic.go:356] "Generic (PLEG): container finished" podID="f86a7659-10d5-4066-b558-c4db7c662362" containerID="976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d" exitCode=0
Oct 30 03:12:50.366113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.365675    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"
Oct 30 03:12:50.366113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.365711    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" event={"ID":"f86a7659-10d5-4066-b558-c4db7c662362","Type":"ContainerDied","Data":"976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d"}
Oct 30 03:12:50.366113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.365775    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc" event={"ID":"f86a7659-10d5-4066-b558-c4db7c662362","Type":"ContainerDied","Data":"5cc1bcad8bf847d442b11c595c027fb38d1f0bcd25c68808cfebdfd55bf77d64"}
Oct 30 03:12:50.366113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.365805    4877 scope.go:122] "RemoveContainer" containerID="976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d"
Oct 30 03:12:50.376988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.376969    4877 scope.go:122] "RemoveContainer" containerID="904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"
Oct 30 03:12:50.386326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.386308    4877 scope.go:122] "RemoveContainer" containerID="976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d"
Oct 30 03:12:50.386598 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:50.386577    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d\": container with ID starting with 976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d not found: ID does not exist" containerID="976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d"
Oct 30 03:12:50.386662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.386608    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d"} err="failed to get container status \"976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d\": rpc error: code = NotFound desc = could not find container \"976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d\": container with ID starting with 976147862a1a7ffdad894bf95f3c76242d8d108d54c7fdc4a0f321b9a269ba4d not found: ID does not exist"
Oct 30 03:12:50.386662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.386625    4877 scope.go:122] "RemoveContainer" containerID="904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"
Oct 30 03:12:50.386855 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:12:50.386835    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a\": container with ID starting with 904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a not found: ID does not exist" containerID="904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"
Oct 30 03:12:50.386915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.386864    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a"} err="failed to get container status \"904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a\": rpc error: code = NotFound desc = could not find container \"904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a\": container with ID starting with 904f2c6ef3cdbe52d9928b9540ef8ea61aeedbb0ed74c004459290a505a2701a not found: ID does not exist"
Oct 30 03:12:50.422650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.422595    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"]
Oct 30 03:12:50.425507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.425477    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3f9206f0-openshift-default-78798cc465-nvfqc"]
Oct 30 03:12:50.427588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.427564    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkxx\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-kube-api-access-9fkxx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.427588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.427588    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f86a7659-10d5-4066-b558-c4db7c662362-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.427736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.427604    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f86a7659-10d5-4066-b558-c4db7c662362-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.427736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.427616    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f86a7659-10d5-4066-b558-c4db7c662362-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:12:50.852327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852296    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"]
Oct 30 03:12:50.852776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852763    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1514bc1c-9d56-48a2-864f-8640bf315dda" containerName="payload-pre-processing"
Oct 30 03:12:50.852822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852778    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1514bc1c-9d56-48a2-864f-8640bf315dda" containerName="payload-pre-processing"
Oct 30 03:12:50.852822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852792    4877 cpu_manager.go:414] "containerMap: removing container" podUID="34a58d2f-2944-4520-a3b0-e6ae65984f2b" containerName="payload-processing"
Oct 30 03:12:50.852822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852797    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a58d2f-2944-4520-a3b0-e6ae65984f2b" containerName="payload-processing"
Oct 30 03:12:50.852822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852810    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy"
Oct 30 03:12:50.852822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852814    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy"
Oct 30 03:12:50.852974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852838    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy"
Oct 30 03:12:50.852974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852844    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy"
Oct 30 03:12:50.852974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852919    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy"
Oct 30 03:12:50.852974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852928    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1514bc1c-9d56-48a2-864f-8640bf315dda" containerName="payload-pre-processing"
Oct 30 03:12:50.852974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.852935    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a58d2f-2944-4520-a3b0-e6ae65984f2b" containerName="payload-processing"
Oct 30 03:12:50.859246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.859223    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:50.874836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.874810    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8d80b1\""
Oct 30 03:12:50.874974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.874915    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8d80b1-dockercfg-gsnsk\""
Oct 30 03:12:50.875244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.875220    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"]
Oct 30 03:12:50.938282 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.938241    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"]
Oct 30 03:12:50.939284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.939253    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86a7659-10d5-4066-b558-c4db7c662362" containerName="istio-proxy"
Oct 30 03:12:50.943582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.943557    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:50.964643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:50.964619    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"]
Oct 30 03:12:51.034539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.034505    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.034687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.034549    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4mm\" (UniqueName: \"kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm\") pod \"payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.136127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.136041    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.136127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.136088    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4mm\" (UniqueName: \"kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm\") pod \"payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.136324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.136174    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.136324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.136200    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrlqs\" (UniqueName: \"kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs\") pod \"payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.136793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.136773    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.155733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.155700    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4mm\" (UniqueName: \"kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm\") pod \"payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.174615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.174583    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:51.241747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.239224    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrlqs\" (UniqueName: \"kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs\") pod \"payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.241747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.239633    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.241747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.240574    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.264065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.264035    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrlqs\" (UniqueName: \"kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs\") pod \"payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.481920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.481892    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"]
Oct 30 03:12:51.483265 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:51.483223    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5a933d_5487_44e3_ac11_855d8bf48846.slice/crio-d063cf7208e04ba2a3555b3d2b8f32544c3ac04bf310e515e81eaf5227d1b0c0 WatchSource:0}: Error finding container d063cf7208e04ba2a3555b3d2b8f32544c3ac04bf310e515e81eaf5227d1b0c0: Status 404 returned error can't find the container with id d063cf7208e04ba2a3555b3d2b8f32544c3ac04bf310e515e81eaf5227d1b0c0
Oct 30 03:12:51.558143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.558116    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:51.699599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.699567    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86a7659-10d5-4066-b558-c4db7c662362" path="/var/lib/kubelet/pods/f86a7659-10d5-4066-b558-c4db7c662362/volumes"
Oct 30 03:12:51.699894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.699879    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"]
Oct 30 03:12:51.705459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.705434    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:51.720385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.720346    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3c874f\""
Oct 30 03:12:51.722730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.722706    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"]
Oct 30 03:12:51.794467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.794408    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3c874f-dockercfg-rh46l\""
Oct 30 03:12:51.795681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.795636    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"]
Oct 30 03:12:51.800389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.800364    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:51.826478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.826451    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"]
Oct 30 03:12:51.844301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.844274    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qsm\" (UniqueName: \"kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm\") pod \"payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:51.844503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.844487    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:51.857057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.857030    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:12:51.857980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.857944    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:12:51.874289 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.874266    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"]
Oct 30 03:12:51.946438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.946392    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qsm\" (UniqueName: \"kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm\") pod \"payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:51.946550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.946478    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:51.946550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.946512    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmjs\" (UniqueName: \"kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs\") pod \"payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:51.946807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.946614    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:51.947198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.947175    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:51.965729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:51.965704    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qsm\" (UniqueName: \"kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm\") pod \"payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:52.019432 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.019388    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:52.047918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.047845    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmjs\" (UniqueName: \"kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs\") pod \"payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:52.048053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.047928    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:52.048561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.048540    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:52.072610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.072586    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmjs\" (UniqueName: \"kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs\") pod \"payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:52.133087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.133042    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:52.325614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.324960    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"]
Oct 30 03:12:52.328094 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:52.328064    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb72dc63_2e16_4bef_beb9_ae5ce2605774.slice/crio-9568d37ca4f1aeb9bb7107464598126ca5e02c2826056b68fbbac9aebf54b0fe WatchSource:0}: Error finding container 9568d37ca4f1aeb9bb7107464598126ca5e02c2826056b68fbbac9aebf54b0fe: Status 404 returned error can't find the container with id 9568d37ca4f1aeb9bb7107464598126ca5e02c2826056b68fbbac9aebf54b0fe
Oct 30 03:12:52.385747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.385713    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" event={"ID":"21fb84a1-a244-4ca4-9b9c-070c2859b235","Type":"ContainerStarted","Data":"b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1"}
Oct 30 03:12:52.385868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.385780    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" event={"ID":"21fb84a1-a244-4ca4-9b9c-070c2859b235","Type":"ContainerStarted","Data":"c38686160d72c84f915f4e32e9c5564eeb06e50ddbc79977e532b5ec7fd414cd"}
Oct 30 03:12:52.385954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.385933    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:12:52.387808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.387781    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" event={"ID":"ac5a933d-5487-44e3-ac11-855d8bf48846","Type":"ContainerStarted","Data":"dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a"}
Oct 30 03:12:52.387921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.387826    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" event={"ID":"ac5a933d-5487-44e3-ac11-855d8bf48846","Type":"ContainerStarted","Data":"d063cf7208e04ba2a3555b3d2b8f32544c3ac04bf310e515e81eaf5227d1b0c0"}
Oct 30 03:12:52.387921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.387892    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:12:52.389940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.389916    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" event={"ID":"cb72dc63-2e16-4bef-beb9-ae5ce2605774","Type":"ContainerStarted","Data":"9568d37ca4f1aeb9bb7107464598126ca5e02c2826056b68fbbac9aebf54b0fe"}
Oct 30 03:12:52.431958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.431907    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" podStartSLOduration=2.431893899 podStartE2EDuration="2.431893899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:52.428129724 +0000 UTC m=+600.857019142" watchObservedRunningTime="2026-10-30 03:12:52.431893899 +0000 UTC m=+600.860783317"
Oct 30 03:12:52.452509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.452477    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"]
Oct 30 03:12:52.453774 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:52.453749    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe2da65_df59_4c30_bb1c_57a33bc91abd.slice/crio-bb269c92dbf2fb9374377ab54fe7624ce9b96fc7fca06a16c1fc069e5333cdee WatchSource:0}: Error finding container bb269c92dbf2fb9374377ab54fe7624ce9b96fc7fca06a16c1fc069e5333cdee: Status 404 returned error can't find the container with id bb269c92dbf2fb9374377ab54fe7624ce9b96fc7fca06a16c1fc069e5333cdee
Oct 30 03:12:52.475134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:52.475078    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" podStartSLOduration=2.475059154 podStartE2EDuration="2.475059154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:52.473523917 +0000 UTC m=+600.902413337" watchObservedRunningTime="2026-10-30 03:12:52.475059154 +0000 UTC m=+600.903948573"
Oct 30 03:12:53.411483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.411445    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" event={"ID":"cb72dc63-2e16-4bef-beb9-ae5ce2605774","Type":"ContainerStarted","Data":"24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4"}
Oct 30 03:12:53.425463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.412885    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:12:53.435843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.435276    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" event={"ID":"cfe2da65-df59-4c30-bb1c-57a33bc91abd","Type":"ContainerStarted","Data":"36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9"}
Oct 30 03:12:53.435843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.435335    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:12:53.435843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.435358    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" event={"ID":"cfe2da65-df59-4c30-bb1c-57a33bc91abd","Type":"ContainerStarted","Data":"bb269c92dbf2fb9374377ab54fe7624ce9b96fc7fca06a16c1fc069e5333cdee"}
Oct 30 03:12:53.454001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.453951    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" podStartSLOduration=2.45393999 podStartE2EDuration="2.45393999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:53.45158804 +0000 UTC m=+601.880477484" watchObservedRunningTime="2026-10-30 03:12:53.45393999 +0000 UTC m=+601.882829408"
Oct 30 03:12:53.506447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:53.500071    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" podStartSLOduration=2.500050412 podStartE2EDuration="2.500050412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:53.492930277 +0000 UTC m=+601.921819696" watchObservedRunningTime="2026-10-30 03:12:53.500050412 +0000 UTC m=+601.928939831"
Oct 30 03:12:54.252553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.252510    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"]
Oct 30 03:12:54.256803 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.256781    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.266256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.266229    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.266327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.266270    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9qs\" (UniqueName: \"kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs\") pod \"payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.275458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.274861    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e44313\""
Oct 30 03:12:54.275458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.274917    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e44313-dockercfg-wst67\""
Oct 30 03:12:54.275458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.275249    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"]
Oct 30 03:12:54.342039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.341999    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"]
Oct 30 03:12:54.352174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.352039    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.368052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.366327    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"]
Oct 30 03:12:54.368052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.367379    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75sn\" (UniqueName: \"kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn\") pod \"payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.368052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.367475    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.368052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.367517    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9qs\" (UniqueName: \"kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs\") pod \"payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.368052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.367553    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.368518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.368494    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.409259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.409227    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9qs\" (UniqueName: \"kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs\") pod \"payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.468326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.468291    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75sn\" (UniqueName: \"kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn\") pod \"payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.468800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.468371    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.469115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.469097    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.505279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.505197    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75sn\" (UniqueName: \"kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn\") pod \"payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.570760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.570724    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:54.729139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.729099    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:54.906634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:54.906602    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"]
Oct 30 03:12:54.907165 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:54.907142    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff8c4ad_1da5_44ac_9f36_75dce02091c4.slice/crio-cc2a46637d2e83e69f0d342aa1094fcc5e75dc9838d6664c048ac8d0b50b638d WatchSource:0}: Error finding container cc2a46637d2e83e69f0d342aa1094fcc5e75dc9838d6664c048ac8d0b50b638d: Status 404 returned error can't find the container with id cc2a46637d2e83e69f0d342aa1094fcc5e75dc9838d6664c048ac8d0b50b638d
Oct 30 03:12:55.070235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.070209    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"]
Oct 30 03:12:55.070938 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:12:55.070911    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f31e566_4b61_40b3_a685_4dd0c4bfce2b.slice/crio-3ae0470d7c32fab0a673ad3bc48929d6ee5798366ff52f81ff79c52faa7a9d01 WatchSource:0}: Error finding container 3ae0470d7c32fab0a673ad3bc48929d6ee5798366ff52f81ff79c52faa7a9d01: Status 404 returned error can't find the container with id 3ae0470d7c32fab0a673ad3bc48929d6ee5798366ff52f81ff79c52faa7a9d01
Oct 30 03:12:55.449235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.449134    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85" event={"ID":"4f31e566-4b61-40b3-a685-4dd0c4bfce2b","Type":"ContainerStarted","Data":"c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742"}
Oct 30 03:12:55.449235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.449197    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85" event={"ID":"4f31e566-4b61-40b3-a685-4dd0c4bfce2b","Type":"ContainerStarted","Data":"3ae0470d7c32fab0a673ad3bc48929d6ee5798366ff52f81ff79c52faa7a9d01"}
Oct 30 03:12:55.449235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.449224    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:12:55.450808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.450780    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2" event={"ID":"1ff8c4ad-1da5-44ac-9f36-75dce02091c4","Type":"ContainerStarted","Data":"b9aac0be858f2db1b1170ebbdac052fe19a1bb435b65f6f5693a91e3ff5a9cdd"}
Oct 30 03:12:55.450966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.450826    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2" event={"ID":"1ff8c4ad-1da5-44ac-9f36-75dce02091c4","Type":"ContainerStarted","Data":"cc2a46637d2e83e69f0d342aa1094fcc5e75dc9838d6664c048ac8d0b50b638d"}
Oct 30 03:12:55.450966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.450862    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:12:55.541131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.541075    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85" podStartSLOduration=1.541060575 podStartE2EDuration="1.541060575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:55.496376288 +0000 UTC m=+603.925265708" watchObservedRunningTime="2026-10-30 03:12:55.541060575 +0000 UTC m=+603.969949993"
Oct 30 03:12:55.542026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:55.541999    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2" podStartSLOduration=1.5419897009999999 podStartE2EDuration="1.541989701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:55.540151898 +0000 UTC m=+603.969041333" watchObservedRunningTime="2026-10-30 03:12:55.541989701 +0000 UTC m=+603.970879118"
Oct 30 03:12:58.345692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:58.345665    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 03:12:58.346136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:12:58.345731    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 03:13:00.372113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:00.372078    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 03:13:00.373467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:00.373444    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 03:13:02.499891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:02.499806    4877 generic.go:356] "Generic (PLEG): container finished" podID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerID="fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877" exitCode=0
Oct 30 03:13:02.499891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:02.499877    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" event={"ID":"713b5c0c-6af2-4b8b-a518-22df5d689146","Type":"ContainerDied","Data":"fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"}
Oct 30 03:13:02.500447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:02.500235    4877 scope.go:122] "RemoveContainer" containerID="fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"
Oct 30 03:13:02.501539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:02.501513    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:02.501614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:02.501575    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:02.501614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:02.501608    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:03.438004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:03.437976    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 03:13:03.438174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:03.438127    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 03:13:03.518860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:03.518827    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" event={"ID":"713b5c0c-6af2-4b8b-a518-22df5d689146","Type":"ContainerStarted","Data":"4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f"}
Oct 30 03:13:04.443099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:04.443066    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 03:13:04.447019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:04.447002    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:13:04.447159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:04.447142    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:13:04.447197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:04.447181    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 03:13:05.447489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:05.447442    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:13:05.448001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:05.447510    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 03:13:05.454579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:05.454553    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 03:13:06.446931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:06.446889    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:13:06.447147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:06.446955    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 03:13:06.462537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:06.462505    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 03:13:06.462934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:06.462788    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 03:13:07.455451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:07.455292    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:13:07.455451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:07.455349    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 03:13:08.446681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:08.446648    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:13:08.447124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:08.446703    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 03:13:09.451226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:09.451199    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:13:09.451673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:09.451652    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:13:09.452446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:09.452408    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:13:15.615991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:15.615953    4877 generic.go:356] "Generic (PLEG): container finished" podID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerID="fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2" exitCode=0
Oct 30 03:13:15.616449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:15.616026    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" event={"ID":"2d09e0b5-a925-4499-973c-85092ecdf66b","Type":"ContainerDied","Data":"fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"}
Oct 30 03:13:15.616499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:15.616455    4877 scope.go:122] "RemoveContainer" containerID="fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"
Oct 30 03:13:15.617682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:15.617651    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:15.617753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:15.617722    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:15.617789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:15.617754    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:13:16.228383 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:13:16.228349    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:13:16.625400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:16.625363    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" event={"ID":"2d09e0b5-a925-4499-973c-85092ecdf66b","Type":"ContainerStarted","Data":"226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a"}
Oct 30 03:13:17.601450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:17.601019    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:13:17.602482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:17.602457    4877 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 03:13:17.602611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:17.602506    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 03:13:18.601446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:18.600835    4877 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 03:13:18.601446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:18.600940    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 03:13:19.119705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.119660    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"]
Oct 30 03:13:19.130463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.128678    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"]
Oct 30 03:13:19.130463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.128789    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.144736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.144713    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3c874f\"/\"kube-root-ca.crt\""
Oct 30 03:13:19.156058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.156033    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3c874f\"/\"default-dockercfg-x4pnr\""
Oct 30 03:13:19.156058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.156035    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3c874f\"/\"openshift-service-ca.crt\""
Oct 30 03:13:19.156215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.156100    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3c874f\"/\"simulator-w3-3c874f-kserve-self-signed-certs\""
Oct 30 03:13:19.227072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.227035    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.227248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.227087    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2r7l\" (UniqueName: \"kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.227248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.227166    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.227372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.227314    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.227416 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.227403    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.227497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.227454    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.328857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.328815    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2r7l\" (UniqueName: \"kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.329155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.329131    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.329329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.329312    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.329515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.329495    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.329663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.329644    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.329826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.329810    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.330467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.330395    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.331507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.331484    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.331885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.331863    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.337140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.337117    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.339098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.339019    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.350254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.350225    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2r7l\" (UniqueName: \"kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l\") pod \"simulator-w3-3c874f-kserve-7f85d45f7-d5ktb\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") " pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.357883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.357861    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:13:19.443694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.443610    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:19.602921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.602852    4877 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 03:13:19.602921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.602909    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 03:13:19.773920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:19.773895    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"]
Oct 30 03:13:19.774071 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:13:19.774049    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08857070_3f09_450e_affb_e7725c6436db.slice/crio-759bc345f632ae93ac285c8821866900cffbaaa077ca1e334e1b625c6ee996f5 WatchSource:0}: Error finding container 759bc345f632ae93ac285c8821866900cffbaaa077ca1e334e1b625c6ee996f5: Status 404 returned error can't find the container with id 759bc345f632ae93ac285c8821866900cffbaaa077ca1e334e1b625c6ee996f5
Oct 30 03:13:20.600446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:20.599855    4877 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 03:13:20.600446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:20.599928    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 03:13:20.656389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:20.656347    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" event={"ID":"08857070-3f09-450e-affb-e7725c6436db","Type":"ContainerStarted","Data":"141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67"}
Oct 30 03:13:20.656856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:20.656400    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" event={"ID":"08857070-3f09-450e-affb-e7725c6436db","Type":"ContainerStarted","Data":"759bc345f632ae93ac285c8821866900cffbaaa077ca1e334e1b625c6ee996f5"}
Oct 30 03:13:20.705442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:20.705363    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podStartSLOduration=1.705331584 podStartE2EDuration="1.705331584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:20.703967391 +0000 UTC m=+629.132856809" watchObservedRunningTime="2026-10-30 03:13:20.705331584 +0000 UTC m=+629.134221003"
Oct 30 03:13:21.444561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:21.444532    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:21.597937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:21.597905    4877 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 03:13:21.598118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:21.597958    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 03:13:22.602989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:22.602953    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:13:22.604375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:22.604350    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:13:29.444646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:29.444608    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:29.457164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:29.457140    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:29.741938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:29.741854    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:13:34.665310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.665271    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"]
Oct 30 03:13:34.672989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.672960    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.693243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.693215    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"openshift-service-ca.crt\""
Oct 30 03:13:34.703919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.703890    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"kube-root-ca.crt\""
Oct 30 03:13:34.704172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.704149    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"simulator-w1-e44313-kserve-self-signed-certs\""
Oct 30 03:13:34.704631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.704611    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"default-dockercfg-cqsbs\""
Oct 30 03:13:34.709295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.709266    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"]
Oct 30 03:13:34.793983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.793945    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.794280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.794239    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9knh\" (UniqueName: \"kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.794543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.794516    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.794640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.794566    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.795140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.795115    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.795225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.795158    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896250    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896312    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896354    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896407    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9knh\" (UniqueName: \"kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896568    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896607    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.896869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.896840    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.897071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.897036    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.897416 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.897395    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.920165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.920041    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.920725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.920702    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.937772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.937742    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9knh\" (UniqueName: \"kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh\") pod \"simulator-w1-e44313-kserve-6f659d4b5b-kmvbf\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") " pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:34.992723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:34.992678    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:35.345878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:35.345848    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"]
Oct 30 03:13:35.347154 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:13:35.347126    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffc107c8_f93e_4124_979e_0a07d6f8df75.slice/crio-d3c84dfad88befae5d688f9817be5336af19228a995ece1fa4c7c6b24e40a681 WatchSource:0}: Error finding container d3c84dfad88befae5d688f9817be5336af19228a995ece1fa4c7c6b24e40a681: Status 404 returned error can't find the container with id d3c84dfad88befae5d688f9817be5336af19228a995ece1fa4c7c6b24e40a681
Oct 30 03:13:35.773409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:35.773375    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" event={"ID":"ffc107c8-f93e-4124-979e-0a07d6f8df75","Type":"ContainerStarted","Data":"e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff"}
Oct 30 03:13:35.773838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:35.773433    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" event={"ID":"ffc107c8-f93e-4124-979e-0a07d6f8df75","Type":"ContainerStarted","Data":"d3c84dfad88befae5d688f9817be5336af19228a995ece1fa4c7c6b24e40a681"}
Oct 30 03:13:35.819583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:35.819518    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podStartSLOduration=1.819500606 podStartE2EDuration="1.819500606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:35.815414203 +0000 UTC m=+644.244303618" watchObservedRunningTime="2026-10-30 03:13:35.819500606 +0000 UTC m=+644.248390025"
Oct 30 03:13:35.994011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:35.993965    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:44.994735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:44.994698    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:45.008462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:45.008417    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:45.858574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:45.858539    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:13:46.228707 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:13:46.228619    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:13:55.601618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.601584    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"]
Oct 30 03:13:55.606955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.606934    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.635687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.635656    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3c874f\"/\"premium-w3-3c874f-kserve-self-signed-certs\""
Oct 30 03:13:55.647156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.647128    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"]
Oct 30 03:13:55.725124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.725086    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.725302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.725250    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.725302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.725293    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpqd\" (UniqueName: \"kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.725406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.725387    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.725506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.725491    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.725549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.725515    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.826597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.826558    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.826597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.826602    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpqd\" (UniqueName: \"kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.826857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.826647    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.826857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.826703    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.826857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.826732    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.826857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.826800    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.827076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.827036    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.827134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.827089    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.827334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.827307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.829141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.829118    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.829782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.829757    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.848834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.848258    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpqd\" (UniqueName: \"kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd\") pod \"premium-w3-3c874f-kserve-5bb78ffb-svs85\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") " pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:55.919823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:55.919720    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:13:56.231965 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:56.231934    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"]
Oct 30 03:13:56.232167 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:13:56.232143    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6849ee53_403b_499c_baa7_a7ccb92b5ff1.slice/crio-14597affd6e8cbaa368ebcf822d76ea9d203a28b2a77cdf72ffbb25b14aebee5 WatchSource:0}: Error finding container 14597affd6e8cbaa368ebcf822d76ea9d203a28b2a77cdf72ffbb25b14aebee5: Status 404 returned error can't find the container with id 14597affd6e8cbaa368ebcf822d76ea9d203a28b2a77cdf72ffbb25b14aebee5
Oct 30 03:13:56.929235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:56.929195    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" event={"ID":"6849ee53-403b-499c-baa7-a7ccb92b5ff1","Type":"ContainerStarted","Data":"b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c"}
Oct 30 03:13:56.929751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:56.929248    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" event={"ID":"6849ee53-403b-499c-baa7-a7ccb92b5ff1","Type":"ContainerStarted","Data":"14597affd6e8cbaa368ebcf822d76ea9d203a28b2a77cdf72ffbb25b14aebee5"}
Oct 30 03:13:56.971622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:56.971567    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podStartSLOduration=1.971552277 podStartE2EDuration="1.971552277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:56.969786016 +0000 UTC m=+665.398675455" watchObservedRunningTime="2026-10-30 03:13:56.971552277 +0000 UTC m=+665.400441695"
Oct 30 03:13:57.919851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:13:57.919816    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:14:05.920448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:05.920338    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:14:05.933394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:05.933365    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:14:06.018684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:06.018653    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:14:07.059128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.059088    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"]
Oct 30 03:14:07.063781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.063759    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.082246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.082202    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"]
Oct 30 03:14:07.090462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.089497    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"premium-w1-e44313-kserve-self-signed-certs\""
Oct 30 03:14:07.262390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.261140    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.262390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.261204    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5k6n\" (UniqueName: \"kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.262390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.261328    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.262390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.261370    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.262390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.261463    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.262390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.261478    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.362790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.362706    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.362790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.362768    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.362796    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.362911    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.363011    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5k6n\" (UniqueName: \"kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.363228    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.363253    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.363294    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.363450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.363388    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.365289 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.365266    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.365419 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.365397    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.382742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.382710    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5k6n\" (UniqueName: \"kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n\") pod \"premium-w1-e44313-kserve-6f74d6f4b5-w75fv\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") " pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.389851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.389823    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:07.703246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:07.702541    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"]
Oct 30 03:14:07.703246 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:07.702856    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ee1492c_d2c2_4e86_818b_514ac3368623.slice/crio-0cc684a72bd9c4bbd318366b78bc520fcb3b86235664259a27752bbb731324a1 WatchSource:0}: Error finding container 0cc684a72bd9c4bbd318366b78bc520fcb3b86235664259a27752bbb731324a1: Status 404 returned error can't find the container with id 0cc684a72bd9c4bbd318366b78bc520fcb3b86235664259a27752bbb731324a1
Oct 30 03:14:08.021605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:08.021566    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" event={"ID":"3ee1492c-d2c2-4e86-818b-514ac3368623","Type":"ContainerStarted","Data":"d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c"}
Oct 30 03:14:08.021809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:08.021621    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" event={"ID":"3ee1492c-d2c2-4e86-818b-514ac3368623","Type":"ContainerStarted","Data":"0cc684a72bd9c4bbd318366b78bc520fcb3b86235664259a27752bbb731324a1"}
Oct 30 03:14:08.066501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:08.066449    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podStartSLOduration=1.06643598 podStartE2EDuration="1.06643598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:08.062849298 +0000 UTC m=+676.491738715" watchObservedRunningTime="2026-10-30 03:14:08.06643598 +0000 UTC m=+676.495325405"
Oct 30 03:14:08.391026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:08.390919    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:16.243868 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:14:16.243821    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:14:16.345620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.345588    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"]
Oct 30 03:14:16.350660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.350643    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.366176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.366116    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.366176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.366173    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x728\" (UniqueName: \"kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.366472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.366303    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.379227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.378600    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"]
Oct 30 03:14:16.385670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.385646    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:16.385670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.385656    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:16.467864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.467836    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.467864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.467874    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x728\" (UniqueName: \"kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.468102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.467921    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.468701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.468671    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.470294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.470274    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.487281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.487251    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x728\" (UniqueName: \"kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728\") pod \"maas-api-revoke-sub-f10201b1-vqxgv\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.670449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.670153    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:16.990816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:16.990785    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"]
Oct 30 03:14:16.992004 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:16.991976    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ceccd54_3780_4001_a332_bb05dd7aaa91.slice/crio-372c844fc9edc960bafa5a97a3c77dba9d383101a85b50f5c3fa055edf3ecb38 WatchSource:0}: Error finding container 372c844fc9edc960bafa5a97a3c77dba9d383101a85b50f5c3fa055edf3ecb38: Status 404 returned error can't find the container with id 372c844fc9edc960bafa5a97a3c77dba9d383101a85b50f5c3fa055edf3ecb38
Oct 30 03:14:17.100491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:17.100458    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv" event={"ID":"2ceccd54-3780-4001-a332-bb05dd7aaa91","Type":"ContainerStarted","Data":"372c844fc9edc960bafa5a97a3c77dba9d383101a85b50f5c3fa055edf3ecb38"}
Oct 30 03:14:17.390280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:17.390256    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:17.402929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:17.402899    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:18.107619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:18.107587    4877 generic.go:356] "Generic (PLEG): container finished" podID="2ceccd54-3780-4001-a332-bb05dd7aaa91" containerID="dd08d98296ff24771d4b8646b7072c1dcfbf731714bcd34547d32818ff4fdf5e" exitCode=0
Oct 30 03:14:18.107794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:18.107642    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv" event={"ID":"2ceccd54-3780-4001-a332-bb05dd7aaa91","Type":"ContainerDied","Data":"dd08d98296ff24771d4b8646b7072c1dcfbf731714bcd34547d32818ff4fdf5e"}
Oct 30 03:14:18.119312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:18.119286    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:14:19.273600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.273577    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:19.301561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.301530    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca\") pod \"2ceccd54-3780-4001-a332-bb05dd7aaa91\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") "
Oct 30 03:14:19.301690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.301636    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728\" (UniqueName: \"kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728\") pod \"2ceccd54-3780-4001-a332-bb05dd7aaa91\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") "
Oct 30 03:14:19.301762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.301746    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca\") pod \"2ceccd54-3780-4001-a332-bb05dd7aaa91\" (UID: \"2ceccd54-3780-4001-a332-bb05dd7aaa91\") "
Oct 30 03:14:19.301868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.301844    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca" pod "2ceccd54-3780-4001-a332-bb05dd7aaa91" (UID: "2ceccd54-3780-4001-a332-bb05dd7aaa91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:19.302140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.302116    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:19.303718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.303695    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca" pod "2ceccd54-3780-4001-a332-bb05dd7aaa91" (UID: "2ceccd54-3780-4001-a332-bb05dd7aaa91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:19.303805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.303749    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728" pod "2ceccd54-3780-4001-a332-bb05dd7aaa91" (UID: "2ceccd54-3780-4001-a332-bb05dd7aaa91"). InnerVolumeSpecName "kube-api-access-8x728". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:19.402999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.402910    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x728\" (UniqueName: \"kubernetes.io/projected/2ceccd54-3780-4001-a332-bb05dd7aaa91-kube-api-access-8x728\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:19.402999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:19.402935    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ceccd54-3780-4001-a332-bb05dd7aaa91-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:20.124707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:20.124667    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv" event={"ID":"2ceccd54-3780-4001-a332-bb05dd7aaa91","Type":"ContainerDied","Data":"372c844fc9edc960bafa5a97a3c77dba9d383101a85b50f5c3fa055edf3ecb38"}
Oct 30 03:14:20.124901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:20.124734    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="372c844fc9edc960bafa5a97a3c77dba9d383101a85b50f5c3fa055edf3ecb38"
Oct 30 03:14:20.124901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:20.124683    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"
Oct 30 03:14:25.630159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.630121    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"]
Oct 30 03:14:25.630686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.630672    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2ceccd54-3780-4001-a332-bb05dd7aaa91" containerName="revoke-keys"
Oct 30 03:14:25.630731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.630687    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ceccd54-3780-4001-a332-bb05dd7aaa91" containerName="revoke-keys"
Oct 30 03:14:25.630798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.630789    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ceccd54-3780-4001-a332-bb05dd7aaa91" containerName="revoke-keys"
Oct 30 03:14:25.634215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.634198    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.661418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.661392    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"]
Oct 30 03:14:25.669517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.669482    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:25.669700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.669681    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:25.703364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.703330    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.703528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.703386    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.703528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.703497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf54t\" (UniqueName: \"kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.805097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.805066    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.805267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.805119    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.805267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.805154    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf54t\" (UniqueName: \"kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.805846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.805827    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.807591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.807567    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.824307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.824286    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf54t\" (UniqueName: \"kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t\") pod \"maas-api-revoke-sub-fb7a3959-zncq9\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:25.947487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:25.947390    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:26.251753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:26.251717    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"]
Oct 30 03:14:26.253081 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:26.253042    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc61acff_9d79_46f0_92e6_652202315377.slice/crio-65a56217e84cff7bc05e96ac83e360c32c47bdbf3ecf795d2d07cbb592331743 WatchSource:0}: Error finding container 65a56217e84cff7bc05e96ac83e360c32c47bdbf3ecf795d2d07cbb592331743: Status 404 returned error can't find the container with id 65a56217e84cff7bc05e96ac83e360c32c47bdbf3ecf795d2d07cbb592331743
Oct 30 03:14:27.177229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:27.177190    4877 generic.go:356] "Generic (PLEG): container finished" podID="fc61acff-9d79-46f0-92e6-652202315377" containerID="811e5301b084e35aa2cd4ab59fa0d1b3ed4a428fd16f3977e2658909ae06f1b9" exitCode=0
Oct 30 03:14:27.177673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:27.177254    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9" event={"ID":"fc61acff-9d79-46f0-92e6-652202315377","Type":"ContainerDied","Data":"811e5301b084e35aa2cd4ab59fa0d1b3ed4a428fd16f3977e2658909ae06f1b9"}
Oct 30 03:14:27.177673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:27.177306    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9" event={"ID":"fc61acff-9d79-46f0-92e6-652202315377","Type":"ContainerStarted","Data":"65a56217e84cff7bc05e96ac83e360c32c47bdbf3ecf795d2d07cbb592331743"}
Oct 30 03:14:28.350173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.350149    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:28.443489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.441212    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca\") pod \"fc61acff-9d79-46f0-92e6-652202315377\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") "
Oct 30 03:14:28.443489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.441326    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t\" (UniqueName: \"kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t\") pod \"fc61acff-9d79-46f0-92e6-652202315377\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") "
Oct 30 03:14:28.443489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.441483    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca\") pod \"fc61acff-9d79-46f0-92e6-652202315377\" (UID: \"fc61acff-9d79-46f0-92e6-652202315377\") "
Oct 30 03:14:28.443489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.442763    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca" pod "fc61acff-9d79-46f0-92e6-652202315377" (UID: "fc61acff-9d79-46f0-92e6-652202315377"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:28.445812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.445781    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca" pod "fc61acff-9d79-46f0-92e6-652202315377" (UID: "fc61acff-9d79-46f0-92e6-652202315377"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:28.446038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.446009    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t" pod "fc61acff-9d79-46f0-92e6-652202315377" (UID: "fc61acff-9d79-46f0-92e6-652202315377"). InnerVolumeSpecName "kube-api-access-xf54t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:28.542836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.542800    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf54t\" (UniqueName: \"kubernetes.io/projected/fc61acff-9d79-46f0-92e6-652202315377-kube-api-access-xf54t\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:28.542836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.542831    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc61acff-9d79-46f0-92e6-652202315377-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:28.542836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:28.542842    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc61acff-9d79-46f0-92e6-652202315377-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:29.191543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:29.191506    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"
Oct 30 03:14:29.191716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:29.191512    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9" event={"ID":"fc61acff-9d79-46f0-92e6-652202315377","Type":"ContainerDied","Data":"65a56217e84cff7bc05e96ac83e360c32c47bdbf3ecf795d2d07cbb592331743"}
Oct 30 03:14:29.191716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:29.191642    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a56217e84cff7bc05e96ac83e360c32c47bdbf3ecf795d2d07cbb592331743"
Oct 30 03:14:32.191507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.191470    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"]
Oct 30 03:14:32.192002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.191988    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fc61acff-9d79-46f0-92e6-652202315377" containerName="revoke-keys"
Oct 30 03:14:32.192002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.192003    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc61acff-9d79-46f0-92e6-652202315377" containerName="revoke-keys"
Oct 30 03:14:32.192106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.192097    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc61acff-9d79-46f0-92e6-652202315377" containerName="revoke-keys"
Oct 30 03:14:32.195506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.195489    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.216296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.216264    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"]
Oct 30 03:14:32.223152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.223127    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:32.223320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.223204    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:32.289230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.289180    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874wg\" (UniqueName: \"kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.289230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.289236    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.289531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.289328    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.390141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.390108    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874wg\" (UniqueName: \"kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.390323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.390157    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.390323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.390200    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.390947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.390924    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.392703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.392683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.409567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.409541    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874wg\" (UniqueName: \"kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg\") pod \"maas-api-revoke-sub-206d9152-86g42\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.510284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.510195    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:32.897540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:32.897508    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"]
Oct 30 03:14:32.900390 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:32.900361    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e6dd3c_f13b_4935_aa2f_9ea53a8a423e.slice/crio-53ad63cf552dcaf437cb7fa6c9d34593b48f7847dbfb74daea83db4f017fddca WatchSource:0}: Error finding container 53ad63cf552dcaf437cb7fa6c9d34593b48f7847dbfb74daea83db4f017fddca: Status 404 returned error can't find the container with id 53ad63cf552dcaf437cb7fa6c9d34593b48f7847dbfb74daea83db4f017fddca
Oct 30 03:14:33.233980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:33.233937    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42" event={"ID":"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e","Type":"ContainerStarted","Data":"4465fc01aa50bde03f51ba938c2eb6e176f2be8da323d81c5e8b9cc3e9e4b22b"}
Oct 30 03:14:33.234475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:33.234016    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42" event={"ID":"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e","Type":"ContainerStarted","Data":"53ad63cf552dcaf437cb7fa6c9d34593b48f7847dbfb74daea83db4f017fddca"}
Oct 30 03:14:33.273634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:33.273581    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42" podStartSLOduration=1.273563025 podStartE2EDuration="1.273563025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:33.272168121 +0000 UTC m=+701.701057557" watchObservedRunningTime="2026-10-30 03:14:33.273563025 +0000 UTC m=+701.702452488"
Oct 30 03:14:34.241374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:34.241341    4877 generic.go:356] "Generic (PLEG): container finished" podID="f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" containerID="4465fc01aa50bde03f51ba938c2eb6e176f2be8da323d81c5e8b9cc3e9e4b22b" exitCode=0
Oct 30 03:14:34.241779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:34.241456    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42" event={"ID":"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e","Type":"ContainerDied","Data":"4465fc01aa50bde03f51ba938c2eb6e176f2be8da323d81c5e8b9cc3e9e4b22b"}
Oct 30 03:14:35.396022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.395993    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:35.527588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.527557    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca\") pod \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") "
Oct 30 03:14:35.527752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.527641    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca\") pod \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") "
Oct 30 03:14:35.527752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.527687    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg\" (UniqueName: \"kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg\") pod \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\" (UID: \"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e\") "
Oct 30 03:14:35.528048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.528019    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca" pod "f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" (UID: "f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:35.528212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.528126    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:35.529879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.529848    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg" pod "f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" (UID: "f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e"). InnerVolumeSpecName "kube-api-access-874wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:35.530102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.530087    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca" pod "f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" (UID: "f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:35.629511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.629476    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874wg\" (UniqueName: \"kubernetes.io/projected/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-kube-api-access-874wg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:35.629511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:35.629515    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:36.255704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:36.255669    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42" event={"ID":"f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e","Type":"ContainerDied","Data":"53ad63cf552dcaf437cb7fa6c9d34593b48f7847dbfb74daea83db4f017fddca"}
Oct 30 03:14:36.255704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:36.255697    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"
Oct 30 03:14:36.255922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:36.255723    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53ad63cf552dcaf437cb7fa6c9d34593b48f7847dbfb74daea83db4f017fddca"
Oct 30 03:14:39.976271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:39.976233    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"]
Oct 30 03:14:39.976907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:39.976888    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" containerName="revoke-keys"
Oct 30 03:14:39.976907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:39.976907    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" containerName="revoke-keys"
Oct 30 03:14:39.977023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:39.977011    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" containerName="revoke-keys"
Oct 30 03:14:39.980451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:39.980435    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:39.998553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:39.998520    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"]
Oct 30 03:14:40.005893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.005869    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:40.006013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.005990    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:40.069028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.068997    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.069191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.069147    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.069191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.069169    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pst\" (UniqueName: \"kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.170279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.170244    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.170493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.170346    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.170493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.170365    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pst\" (UniqueName: \"kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.171039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.171013    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.172608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.172587    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.189526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.189498    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pst\" (UniqueName: \"kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst\") pod \"maas-api-revoke-sub-fc6bb64e-rm84g\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.293360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.293321    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:40.603848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:40.603817    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"]
Oct 30 03:14:40.606115 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:40.606066    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd99b12e_3fe4_445d_b891_6a69f114018a.slice/crio-927e0155461e64517113bc99e8baca7f9797aeed21aa7e19ddd4d4bdf4043e33 WatchSource:0}: Error finding container 927e0155461e64517113bc99e8baca7f9797aeed21aa7e19ddd4d4bdf4043e33: Status 404 returned error can't find the container with id 927e0155461e64517113bc99e8baca7f9797aeed21aa7e19ddd4d4bdf4043e33
Oct 30 03:14:41.296457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:41.296400    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g" event={"ID":"cd99b12e-3fe4-445d-b891-6a69f114018a","Type":"ContainerStarted","Data":"9aebdb17474efa825948493070c96c85d2c0281a63707e4c54398ad500170529"}
Oct 30 03:14:41.296909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:41.296475    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g" event={"ID":"cd99b12e-3fe4-445d-b891-6a69f114018a","Type":"ContainerStarted","Data":"927e0155461e64517113bc99e8baca7f9797aeed21aa7e19ddd4d4bdf4043e33"}
Oct 30 03:14:41.338977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:41.338924    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g" podStartSLOduration=2.338910541 podStartE2EDuration="2.338910541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:41.336036167 +0000 UTC m=+709.764925590" watchObservedRunningTime="2026-10-30 03:14:41.338910541 +0000 UTC m=+709.767799959"
Oct 30 03:14:42.307170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:42.307133    4877 generic.go:356] "Generic (PLEG): container finished" podID="cd99b12e-3fe4-445d-b891-6a69f114018a" containerID="9aebdb17474efa825948493070c96c85d2c0281a63707e4c54398ad500170529" exitCode=0
Oct 30 03:14:42.307645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:42.307221    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g" event={"ID":"cd99b12e-3fe4-445d-b891-6a69f114018a","Type":"ContainerDied","Data":"9aebdb17474efa825948493070c96c85d2c0281a63707e4c54398ad500170529"}
Oct 30 03:14:43.482614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.482585    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:43.541861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.541824    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca\") pod \"cd99b12e-3fe4-445d-b891-6a69f114018a\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") "
Oct 30 03:14:43.542059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.541970    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst\" (UniqueName: \"kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst\") pod \"cd99b12e-3fe4-445d-b891-6a69f114018a\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") "
Oct 30 03:14:43.542059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.542045    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca\") pod \"cd99b12e-3fe4-445d-b891-6a69f114018a\" (UID: \"cd99b12e-3fe4-445d-b891-6a69f114018a\") "
Oct 30 03:14:43.544748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.544717    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca" pod "cd99b12e-3fe4-445d-b891-6a69f114018a" (UID: "cd99b12e-3fe4-445d-b891-6a69f114018a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:43.545047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.545015    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca" pod "cd99b12e-3fe4-445d-b891-6a69f114018a" (UID: "cd99b12e-3fe4-445d-b891-6a69f114018a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:43.545617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.545588    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst" pod "cd99b12e-3fe4-445d-b891-6a69f114018a" (UID: "cd99b12e-3fe4-445d-b891-6a69f114018a"). InnerVolumeSpecName "kube-api-access-24pst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:43.643619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.643511    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pst\" (UniqueName: \"kubernetes.io/projected/cd99b12e-3fe4-445d-b891-6a69f114018a-kube-api-access-24pst\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:43.643619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.643554    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:43.643619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:43.643568    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd99b12e-3fe4-445d-b891-6a69f114018a-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:44.323534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:44.323496    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"
Oct 30 03:14:44.323753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:44.323495    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g" event={"ID":"cd99b12e-3fe4-445d-b891-6a69f114018a","Type":"ContainerDied","Data":"927e0155461e64517113bc99e8baca7f9797aeed21aa7e19ddd4d4bdf4043e33"}
Oct 30 03:14:44.323753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:44.323611    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927e0155461e64517113bc99e8baca7f9797aeed21aa7e19ddd4d4bdf4043e33"
Oct 30 03:14:46.228895 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:14:46.228858    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:14:46.808953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.808918    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"]
Oct 30 03:14:46.809499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.809484    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cd99b12e-3fe4-445d-b891-6a69f114018a" containerName="revoke-keys"
Oct 30 03:14:46.809577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.809500    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd99b12e-3fe4-445d-b891-6a69f114018a" containerName="revoke-keys"
Oct 30 03:14:46.809649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.809626    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd99b12e-3fe4-445d-b891-6a69f114018a" containerName="revoke-keys"
Oct 30 03:14:46.813353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.813331    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.831602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.831577    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"]
Oct 30 03:14:46.839672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.839642    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:46.839804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.839726    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:46.878014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.877979    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpw9d\" (UniqueName: \"kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.878199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.878025    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.878269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.878191    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.979721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.979681    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.979933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.979791    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.979933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.979877    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw9d\" (UniqueName: \"kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.980572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.980548    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.982088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.982065    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:46.999503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:46.999475    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw9d\" (UniqueName: \"kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d\") pod \"maas-api-revoke-sub-ddf00eb2-rxv4f\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:47.127045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:47.126962    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:47.431143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:47.431108    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"]
Oct 30 03:14:47.431565 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:47.431535    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85f271e_f98f_4c06_96fe_efb267c92169.slice/crio-9dbcbe2bf36cdc0772f626cd2da01a6cd4b6c4d429534e47863c4c611934a831 WatchSource:0}: Error finding container 9dbcbe2bf36cdc0772f626cd2da01a6cd4b6c4d429534e47863c4c611934a831: Status 404 returned error can't find the container with id 9dbcbe2bf36cdc0772f626cd2da01a6cd4b6c4d429534e47863c4c611934a831
Oct 30 03:14:48.351895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:48.351858    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f" event={"ID":"c85f271e-f98f-4c06-96fe-efb267c92169","Type":"ContainerStarted","Data":"a886faff3dd2b992ef5e40b5a24a7d898c87a156c9397341ec2067acf180e16c"}
Oct 30 03:14:48.352068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:48.351917    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f" event={"ID":"c85f271e-f98f-4c06-96fe-efb267c92169","Type":"ContainerStarted","Data":"9dbcbe2bf36cdc0772f626cd2da01a6cd4b6c4d429534e47863c4c611934a831"}
Oct 30 03:14:48.390161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:48.390120    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f" podStartSLOduration=2.390105046 podStartE2EDuration="2.390105046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:48.389075768 +0000 UTC m=+716.817965167" watchObservedRunningTime="2026-10-30 03:14:48.390105046 +0000 UTC m=+716.818994464"
Oct 30 03:14:49.363006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:49.362965    4877 generic.go:356] "Generic (PLEG): container finished" podID="c85f271e-f98f-4c06-96fe-efb267c92169" containerID="a886faff3dd2b992ef5e40b5a24a7d898c87a156c9397341ec2067acf180e16c" exitCode=0
Oct 30 03:14:49.363532 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:49.363052    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f" event={"ID":"c85f271e-f98f-4c06-96fe-efb267c92169","Type":"ContainerDied","Data":"a886faff3dd2b992ef5e40b5a24a7d898c87a156c9397341ec2067acf180e16c"}
Oct 30 03:14:50.540692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.540666    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:50.622869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.621807    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca\") pod \"c85f271e-f98f-4c06-96fe-efb267c92169\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") "
Oct 30 03:14:50.622869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.621910    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca\") pod \"c85f271e-f98f-4c06-96fe-efb267c92169\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") "
Oct 30 03:14:50.622869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.622056    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d\" (UniqueName: \"kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d\") pod \"c85f271e-f98f-4c06-96fe-efb267c92169\" (UID: \"c85f271e-f98f-4c06-96fe-efb267c92169\") "
Oct 30 03:14:50.622869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.622562    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca" pod "c85f271e-f98f-4c06-96fe-efb267c92169" (UID: "c85f271e-f98f-4c06-96fe-efb267c92169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:50.628455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.624559    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca" pod "c85f271e-f98f-4c06-96fe-efb267c92169" (UID: "c85f271e-f98f-4c06-96fe-efb267c92169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:50.640782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.640742    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d" pod "c85f271e-f98f-4c06-96fe-efb267c92169" (UID: "c85f271e-f98f-4c06-96fe-efb267c92169"). InnerVolumeSpecName "kube-api-access-qpw9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:50.723415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.723382    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:50.723415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.723409    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpw9d\" (UniqueName: \"kubernetes.io/projected/c85f271e-f98f-4c06-96fe-efb267c92169-kube-api-access-qpw9d\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:50.723415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:50.723432    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85f271e-f98f-4c06-96fe-efb267c92169-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:51.379344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.379307    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f" event={"ID":"c85f271e-f98f-4c06-96fe-efb267c92169","Type":"ContainerDied","Data":"9dbcbe2bf36cdc0772f626cd2da01a6cd4b6c4d429534e47863c4c611934a831"}
Oct 30 03:14:51.379543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.379371    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dbcbe2bf36cdc0772f626cd2da01a6cd4b6c4d429534e47863c4c611934a831"
Oct 30 03:14:51.379543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.379318    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"
Oct 30 03:14:51.583650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.583614    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"]
Oct 30 03:14:51.584461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.584411    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c85f271e-f98f-4c06-96fe-efb267c92169" containerName="revoke-keys"
Oct 30 03:14:51.584461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.584455    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85f271e-f98f-4c06-96fe-efb267c92169" containerName="revoke-keys"
Oct 30 03:14:51.584685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.584669    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85f271e-f98f-4c06-96fe-efb267c92169" containerName="revoke-keys"
Oct 30 03:14:51.588857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.588835    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.605216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.605194    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:51.607618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.607575    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:51.611101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.611074    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"]
Oct 30 03:14:51.740673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.740588    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rqn\" (UniqueName: \"kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.740673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.740645    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.740901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.740771    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.841589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.841539    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rqn\" (UniqueName: \"kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.841773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.841625    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.841773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.841697    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.842311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.842290    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.855382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.855360    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:14:51.861280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.861252    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rqn\" (UniqueName: \"kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.864043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.864019    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ecb50c-4zft5\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:51.921672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.921635    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:14:51.928859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:51.928832    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:52.232030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:52.231996    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"]
Oct 30 03:14:52.232688 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:52.232653    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f628a5d_93fe_491f_91cc_5d289264d75e.slice/crio-ed3946a0e5d9003c8940f1f5cf2e3e89e25a084ff2aedc0ba27d9aefa0bcf97a WatchSource:0}: Error finding container ed3946a0e5d9003c8940f1f5cf2e3e89e25a084ff2aedc0ba27d9aefa0bcf97a: Status 404 returned error can't find the container with id ed3946a0e5d9003c8940f1f5cf2e3e89e25a084ff2aedc0ba27d9aefa0bcf97a
Oct 30 03:14:52.390692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:52.390654    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5" event={"ID":"6f628a5d-93fe-491f-91cc-5d289264d75e","Type":"ContainerStarted","Data":"ed3946a0e5d9003c8940f1f5cf2e3e89e25a084ff2aedc0ba27d9aefa0bcf97a"}
Oct 30 03:14:53.399724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:53.399688    4877 generic.go:356] "Generic (PLEG): container finished" podID="6f628a5d-93fe-491f-91cc-5d289264d75e" containerID="4d699ca5e010fb97969d4ae8d4b4c6f41d44c247969e14c7e89f7e109717edde" exitCode=0
Oct 30 03:14:53.400141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:53.399775    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5" event={"ID":"6f628a5d-93fe-491f-91cc-5d289264d75e","Type":"ContainerDied","Data":"4d699ca5e010fb97969d4ae8d4b4c6f41d44c247969e14c7e89f7e109717edde"}
Oct 30 03:14:54.570677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.570652    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:54.671651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.671554    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn\" (UniqueName: \"kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn\") pod \"6f628a5d-93fe-491f-91cc-5d289264d75e\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") "
Oct 30 03:14:54.671651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.671608    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca\") pod \"6f628a5d-93fe-491f-91cc-5d289264d75e\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") "
Oct 30 03:14:54.671895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.671743    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca\") pod \"6f628a5d-93fe-491f-91cc-5d289264d75e\" (UID: \"6f628a5d-93fe-491f-91cc-5d289264d75e\") "
Oct 30 03:14:54.671984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.671957    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca" pod "6f628a5d-93fe-491f-91cc-5d289264d75e" (UID: "6f628a5d-93fe-491f-91cc-5d289264d75e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:14:54.672179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.672149    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:54.673729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.673707    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn" pod "6f628a5d-93fe-491f-91cc-5d289264d75e" (UID: "6f628a5d-93fe-491f-91cc-5d289264d75e"). InnerVolumeSpecName "kube-api-access-j8rqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:14:54.674056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.674030    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca" pod "6f628a5d-93fe-491f-91cc-5d289264d75e" (UID: "6f628a5d-93fe-491f-91cc-5d289264d75e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:14:54.772991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.772953    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rqn\" (UniqueName: \"kubernetes.io/projected/6f628a5d-93fe-491f-91cc-5d289264d75e-kube-api-access-j8rqn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:54.772991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:54.772985    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f628a5d-93fe-491f-91cc-5d289264d75e-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:14:55.423171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:55.423141    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"
Oct 30 03:14:55.423372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:55.423144    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5" event={"ID":"6f628a5d-93fe-491f-91cc-5d289264d75e","Type":"ContainerDied","Data":"ed3946a0e5d9003c8940f1f5cf2e3e89e25a084ff2aedc0ba27d9aefa0bcf97a"}
Oct 30 03:14:55.423372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:55.423271    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3946a0e5d9003c8940f1f5cf2e3e89e25a084ff2aedc0ba27d9aefa0bcf97a"
Oct 30 03:14:57.924807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.924763    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"]
Oct 30 03:14:57.925358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.925330    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6f628a5d-93fe-491f-91cc-5d289264d75e" containerName="revoke-keys"
Oct 30 03:14:57.925358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.925342    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f628a5d-93fe-491f-91cc-5d289264d75e" containerName="revoke-keys"
Oct 30 03:14:57.925503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.925448    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f628a5d-93fe-491f-91cc-5d289264d75e" containerName="revoke-keys"
Oct 30 03:14:57.929096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.929073    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:57.943812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.943781    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2d5b09-openshift-default-dockercfg-hpf84\""
Oct 30 03:14:57.966144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:57.966112    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"]
Oct 30 03:14:58.006887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.006850    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.006887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.006890    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hmw2\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.006931    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.007101    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.007177    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.007278    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.007329    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.007376    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.007744 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.007486    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108522    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108590    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108606    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hmw2\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108639    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108696    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108760    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108795    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108823    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.108956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108865    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.109166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108977    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.109166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.108987    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.109320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.109302    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.109375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.109329    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.109681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.109662    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.111714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.111691    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.111987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.111967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.127900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.127867    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.128413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.128394    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hmw2\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.242147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.242068    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:14:58.567098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.567066    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"]
Oct 30 03:14:58.567862 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:58.567837    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b5a35b_1de8_4afe_8dd5_309dd8a95efe.slice/crio-7c20052202ab9e564145fd4c70f6906550ac806113635f20c398b4efb93badd4 WatchSource:0}: Error finding container 7c20052202ab9e564145fd4c70f6906550ac806113635f20c398b4efb93badd4: Status 404 returned error can't find the container with id 7c20052202ab9e564145fd4c70f6906550ac806113635f20c398b4efb93badd4
Oct 30 03:14:58.570634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.570604    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:14:58.570727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.570666    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:14:58.570727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.570696    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:14:58.894736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.894658    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"]
Oct 30 03:14:58.899676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.899654    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:58.919338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.919308    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"]
Oct 30 03:14:58.925647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:58.925615    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d80b1-openshift-default-dockercfg-22948\""
Oct 30 03:14:59.019126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019072    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019134    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019276    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019403    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019477    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxr99\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019579    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019621    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019640    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.019738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.019708    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.120669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120626    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.120854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120763    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.120854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120792    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxr99\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.120854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120845    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.121045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120883    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.121045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120916    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.121045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.120945    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.121045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.121009    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.121045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.121041    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.124195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.122841    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.124195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.123461    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.124195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.123969    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.124324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.124216    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.127355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.125292    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.130589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.130559    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.130989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.130967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.140226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.140197    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.142522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.142490    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxr99\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.214714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.214620    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:14:59.452772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.452731    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" event={"ID":"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe","Type":"ContainerStarted","Data":"e16320c475f095385e86a29b52800aa6aeae8d9af2154e1a90bb66f9ab1c2e5e"}
Oct 30 03:14:59.452927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.452793    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" event={"ID":"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe","Type":"ContainerStarted","Data":"7c20052202ab9e564145fd4c70f6906550ac806113635f20c398b4efb93badd4"}
Oct 30 03:14:59.548345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.548288    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podStartSLOduration=2.548267891 podStartE2EDuration="2.548267891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:59.503834779 +0000 UTC m=+727.932724223" watchObservedRunningTime="2026-10-30 03:14:59.548267891 +0000 UTC m=+727.977157312"
Oct 30 03:14:59.550138 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:14:59.550109    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dfaff2_04a1_4e93_828f_6ce9fbc495e5.slice/crio-d0cacd9ffb7c8abaf897673e44bd124dd49ad3bf59a4dfe93fb049a1d8d3fbfa WatchSource:0}: Error finding container d0cacd9ffb7c8abaf897673e44bd124dd49ad3bf59a4dfe93fb049a1d8d3fbfa: Status 404 returned error can't find the container with id d0cacd9ffb7c8abaf897673e44bd124dd49ad3bf59a4dfe93fb049a1d8d3fbfa
Oct 30 03:14:59.552106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.552083    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"]
Oct 30 03:14:59.552599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.552572    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:14:59.552658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.552630    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:14:59.552700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:14:59.552658    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:15:00.150969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.150935    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"]
Oct 30 03:15:00.155982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.155963    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.156216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.156201    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"]
Oct 30 03:15:00.160256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.160238    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.160870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.160849    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"]
Oct 30 03:15:00.165307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.165290    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.248474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.233237    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:15:00.251187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.251118    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:15:00.253785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.253750    4877 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:15:00.253915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.253813    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 03:15:00.257573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.257549    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"]
Oct 30 03:15:00.261187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.261167    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"]
Oct 30 03:15:00.264588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.264569    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"]
Oct 30 03:15:00.352103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.352060    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888835-7pxsd\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.352284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.352168    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.352284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.352202    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmc8c\" (UniqueName: \"kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.352284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.352227    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cl2w\" (UniqueName: \"kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.352284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.352248    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvf5\" (UniqueName: \"kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5\") pod \"maas-api-key-cleanup-29888835-7pxsd\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.352284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.352279    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.453819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.453723    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvf5\" (UniqueName: \"kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5\") pod \"maas-api-key-cleanup-29888835-7pxsd\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.453819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.453774    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.453819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.453818    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888835-7pxsd\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.454111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.453955    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.454111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.454008    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmc8c\" (UniqueName: \"kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.454215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.454120    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cl2w\" (UniqueName: \"kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.454689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.454667    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.454802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.454711    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.454802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.454713    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888835-7pxsd\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.461778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.461749    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" event={"ID":"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5","Type":"ContainerStarted","Data":"f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710"}
Oct 30 03:15:00.461875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.461799    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" event={"ID":"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5","Type":"ContainerStarted","Data":"d0cacd9ffb7c8abaf897673e44bd124dd49ad3bf59a4dfe93fb049a1d8d3fbfa"}
Oct 30 03:15:00.474298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.474271    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvf5\" (UniqueName: \"kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5\") pod \"maas-api-key-cleanup-29888835-7pxsd\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.474685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.474661    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmc8c\" (UniqueName: \"kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.475995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.475959    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cl2w\" (UniqueName: \"kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.554257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.554027    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:00.563345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.563324    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:00.774349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.774318    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:00.953780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.953716    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podStartSLOduration=2.953650835 podStartE2EDuration="2.953650835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:00.514757729 +0000 UTC m=+728.943647148" watchObservedRunningTime="2026-10-30 03:15:00.953650835 +0000 UTC m=+729.382540254"
Oct 30 03:15:00.954503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:00.954471    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"]
Oct 30 03:15:00.955630 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:15:00.955593    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2cf0f1_23f7_4238_a94d_588d45531df5.slice/crio-1d48d35ffe094b90e0848e5270071501d16116137f77227200c9d7f49275696d WatchSource:0}: Error finding container 1d48d35ffe094b90e0848e5270071501d16116137f77227200c9d7f49275696d: Status 404 returned error can't find the container with id 1d48d35ffe094b90e0848e5270071501d16116137f77227200c9d7f49275696d
Oct 30 03:15:01.032960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.032912    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"]
Oct 30 03:15:01.041160 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:15:01.041102    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78e26d3_3441_431a_8654_94ecd87da441.slice/crio-7088a3e5ca977f57ee3288b78408b1a1342fdd19a908c6df0af3473d47809fe2 WatchSource:0}: Error finding container 7088a3e5ca977f57ee3288b78408b1a1342fdd19a908c6df0af3473d47809fe2: Status 404 returned error can't find the container with id 7088a3e5ca977f57ee3288b78408b1a1342fdd19a908c6df0af3473d47809fe2
Oct 30 03:15:01.224469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.220550    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:15:01.224469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.220913    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:15:01.224469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.220962    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 03:15:01.238772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.235794    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"]
Oct 30 03:15:01.243210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.243177    4877 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:15:01.243332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.243220    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 03:15:01.247716 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:15:01.247692    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3714134a_abab_4e93_a271_ae085eec9bc2.slice/crio-196471e7e22c8ef037beeeead68243a98fe28e371c1dc5a976649bb87b83eecf WatchSource:0}: Error finding container 196471e7e22c8ef037beeeead68243a98fe28e371c1dc5a976649bb87b83eecf: Status 404 returned error can't find the container with id 196471e7e22c8ef037beeeead68243a98fe28e371c1dc5a976649bb87b83eecf
Oct 30 03:15:01.472480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.472397    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerStarted","Data":"fc2e3c255f049114c638d690ba37b81f7e9132a4de4c884e5431f08fca2cdd0a"}
Oct 30 03:15:01.472480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.472471    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerStarted","Data":"1d48d35ffe094b90e0848e5270071501d16116137f77227200c9d7f49275696d"}
Oct 30 03:15:01.474438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.474396    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" event={"ID":"3714134a-abab-4e93-a271-ae085eec9bc2","Type":"ContainerStarted","Data":"196471e7e22c8ef037beeeead68243a98fe28e371c1dc5a976649bb87b83eecf"}
Oct 30 03:15:01.475581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.475560    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" event={"ID":"e78e26d3-3441-431a-8654-94ecd87da441","Type":"ContainerStarted","Data":"7088a3e5ca977f57ee3288b78408b1a1342fdd19a908c6df0af3473d47809fe2"}
Oct 30 03:15:01.536478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:01.536398    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" podStartSLOduration=1.536378046 podStartE2EDuration="1.536378046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:01.532765387 +0000 UTC m=+729.961654826" watchObservedRunningTime="2026-10-30 03:15:01.536378046 +0000 UTC m=+729.965267465"
Oct 30 03:15:02.050431 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.050381    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"]
Oct 30 03:15:02.066080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.065023    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.072910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.071715    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.072910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.071857    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6q4\" (UniqueName: \"kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.072910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.071913    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.072910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.072511    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"]
Oct 30 03:15:02.092655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.091491    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:15:02.173459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.172907    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.173459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.173003    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6q4\" (UniqueName: \"kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.173459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.173035    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.174051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.174031    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.175399    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.200448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.194568    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6q4\" (UniqueName: \"kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4\") pod \"maas-api-revoke-sub-6ba7fa48-sc4wt\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.223449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.215995    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:15:02.223449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.216054    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 03:15:02.242989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.242613    4877 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:15:02.242989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.242667    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 03:15:02.408926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.406039    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:02.544451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.541062    4877 generic.go:356] "Generic (PLEG): container finished" podID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerID="fc2e3c255f049114c638d690ba37b81f7e9132a4de4c884e5431f08fca2cdd0a" exitCode=60
Oct 30 03:15:02.544451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.541199    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerDied","Data":"fc2e3c255f049114c638d690ba37b81f7e9132a4de4c884e5431f08fca2cdd0a"}
Oct 30 03:15:02.544451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.541693    4877 scope.go:122] "RemoveContainer" containerID="fc2e3c255f049114c638d690ba37b81f7e9132a4de4c884e5431f08fca2cdd0a"
Oct 30 03:15:02.547264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.547240    4877 generic.go:356] "Generic (PLEG): container finished" podID="3714134a-abab-4e93-a271-ae085eec9bc2" containerID="8ac65845a09e4a9a6d3d84c781b20a50de92c57ed2713bbeb7f258ca9a4be69f" exitCode=60
Oct 30 03:15:02.547370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.547316    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" event={"ID":"3714134a-abab-4e93-a271-ae085eec9bc2","Type":"ContainerDied","Data":"8ac65845a09e4a9a6d3d84c781b20a50de92c57ed2713bbeb7f258ca9a4be69f"}
Oct 30 03:15:02.547717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.547699    4877 scope.go:122] "RemoveContainer" containerID="8ac65845a09e4a9a6d3d84c781b20a50de92c57ed2713bbeb7f258ca9a4be69f"
Oct 30 03:15:02.550013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.549984    4877 generic.go:356] "Generic (PLEG): container finished" podID="e78e26d3-3441-431a-8654-94ecd87da441" containerID="faa7bab03e7bf16e6403e3fba7722daa38e1d654a2c9597f0b72de2d3ec454ca" exitCode=60
Oct 30 03:15:02.550818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.550796    4877 scope.go:122] "RemoveContainer" containerID="faa7bab03e7bf16e6403e3fba7722daa38e1d654a2c9597f0b72de2d3ec454ca"
Oct 30 03:15:02.550928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.550821    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" event={"ID":"e78e26d3-3441-431a-8654-94ecd87da441","Type":"ContainerDied","Data":"faa7bab03e7bf16e6403e3fba7722daa38e1d654a2c9597f0b72de2d3ec454ca"}
Oct 30 03:15:02.839519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:02.839494    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"]
Oct 30 03:15:03.218270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.217833    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:15:03.218270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.217895    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 03:15:03.242537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.242503    4877 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:15:03.242683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.242565    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 03:15:03.558495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.558449    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt" event={"ID":"f884eb0a-68a1-4fc5-b40d-af25919511b3","Type":"ContainerStarted","Data":"78b95c0647562ac7bdc02511a74473277b1832a9c300a8b8e4547a9420d54a37"}
Oct 30 03:15:03.558896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.558515    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt" event={"ID":"f884eb0a-68a1-4fc5-b40d-af25919511b3","Type":"ContainerStarted","Data":"2c8e72bf17da3a7f6ab55fb92f44250c254274aa287e0d827750e137f3e43a79"}
Oct 30 03:15:03.560639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.560619    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerStarted","Data":"cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b"}
Oct 30 03:15:03.562829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.562806    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" event={"ID":"3714134a-abab-4e93-a271-ae085eec9bc2","Type":"ContainerStarted","Data":"4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351"}
Oct 30 03:15:03.564854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.564835    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" event={"ID":"e78e26d3-3441-431a-8654-94ecd87da441","Type":"ContainerStarted","Data":"913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422"}
Oct 30 03:15:03.644035 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.637776    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt" podStartSLOduration=1.637752433 podStartE2EDuration="1.637752433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.599838222 +0000 UTC m=+732.028727640" watchObservedRunningTime="2026-10-30 03:15:03.637752433 +0000 UTC m=+732.066641852"
Oct 30 03:15:03.651857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.651799    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" podStartSLOduration=3.651784359 podStartE2EDuration="3.651784359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.648961583 +0000 UTC m=+732.077851002" watchObservedRunningTime="2026-10-30 03:15:03.651784359 +0000 UTC m=+732.080673776"
Oct 30 03:15:03.708451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:03.688298    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" podStartSLOduration=3.688275936 podStartE2EDuration="3.688275936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:03.686624844 +0000 UTC m=+732.115514287" watchObservedRunningTime="2026-10-30 03:15:03.688275936 +0000 UTC m=+732.117165355"
Oct 30 03:15:04.225510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.215902    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:15:04.225510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.215969    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 03:15:04.243524 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.243486    4877 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 03:15:04.243704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.243546    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 03:15:04.575959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.575924    4877 generic.go:356] "Generic (PLEG): container finished" podID="f884eb0a-68a1-4fc5-b40d-af25919511b3" containerID="78b95c0647562ac7bdc02511a74473277b1832a9c300a8b8e4547a9420d54a37" exitCode=0
Oct 30 03:15:04.576323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.576018    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt" event={"ID":"f884eb0a-68a1-4fc5-b40d-af25919511b3","Type":"ContainerDied","Data":"78b95c0647562ac7bdc02511a74473277b1832a9c300a8b8e4547a9420d54a37"}
Oct 30 03:15:04.606447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.601558    4877 generic.go:356] "Generic (PLEG): container finished" podID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerID="cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b" exitCode=60
Oct 30 03:15:04.606447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.601729    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerDied","Data":"cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b"}
Oct 30 03:15:04.606447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.601789    4877 scope.go:122] "RemoveContainer" containerID="fc2e3c255f049114c638d690ba37b81f7e9132a4de4c884e5431f08fca2cdd0a"
Oct 30 03:15:04.606447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.602281    4877 scope.go:122] "RemoveContainer" containerID="cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b"
Oct 30 03:15:04.606447 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:04.602638    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888835-7pxsd_odh-ai-gateway-infra(de2cf0f1-23f7-4238-a94d-588d45531df5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5"
Oct 30 03:15:04.628348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.628308    4877 generic.go:356] "Generic (PLEG): container finished" podID="3714134a-abab-4e93-a271-ae085eec9bc2" containerID="4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351" exitCode=60
Oct 30 03:15:04.628692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.628658    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" event={"ID":"3714134a-abab-4e93-a271-ae085eec9bc2","Type":"ContainerDied","Data":"4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351"}
Oct 30 03:15:04.628928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.628879    4877 scope.go:122] "RemoveContainer" containerID="4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351"
Oct 30 03:15:04.629487 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:04.629455    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm_odh-ai-gateway-infra(3714134a-abab-4e93-a271-ae085eec9bc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" podUID="3714134a-abab-4e93-a271-ae085eec9bc2"
Oct 30 03:15:04.634363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.634336    4877 generic.go:356] "Generic (PLEG): container finished" podID="e78e26d3-3441-431a-8654-94ecd87da441" containerID="913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422" exitCode=60
Oct 30 03:15:04.634495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.634477    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" event={"ID":"e78e26d3-3441-431a-8654-94ecd87da441","Type":"ContainerDied","Data":"913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422"}
Oct 30 03:15:04.634695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.634673    4877 scope.go:122] "RemoveContainer" containerID="913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422"
Oct 30 03:15:04.635006 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:04.634985    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm_odh-ai-gateway-infra(e78e26d3-3441-431a-8654-94ecd87da441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" podUID="e78e26d3-3441-431a-8654-94ecd87da441"
Oct 30 03:15:04.641162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.641143    4877 scope.go:122] "RemoveContainer" containerID="8ac65845a09e4a9a6d3d84c781b20a50de92c57ed2713bbeb7f258ca9a4be69f"
Oct 30 03:15:04.654836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:04.654815    4877 scope.go:122] "RemoveContainer" containerID="faa7bab03e7bf16e6403e3fba7722daa38e1d654a2c9597f0b72de2d3ec454ca"
Oct 30 03:15:05.215455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.215260    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 03:15:05.215455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.215351    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 03:15:05.549061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.549021    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:15:05.550477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.550447    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:15:05.551124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.551101    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:15:05.643627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.643596    4877 scope.go:122] "RemoveContainer" containerID="cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b"
Oct 30 03:15:05.644105 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:05.643915    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888835-7pxsd_odh-ai-gateway-infra(de2cf0f1-23f7-4238-a94d-588d45531df5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5"
Oct 30 03:15:05.645635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.645610    4877 scope.go:122] "RemoveContainer" containerID="4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351"
Oct 30 03:15:05.645929 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:05.645903    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm_odh-ai-gateway-infra(3714134a-abab-4e93-a271-ae085eec9bc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" podUID="3714134a-abab-4e93-a271-ae085eec9bc2"
Oct 30 03:15:05.648482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.648018    4877 scope.go:122] "RemoveContainer" containerID="913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422"
Oct 30 03:15:05.648608 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:05.648533    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm_odh-ai-gateway-infra(e78e26d3-3441-431a-8654-94ecd87da441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" podUID="e78e26d3-3441-431a-8654-94ecd87da441"
Oct 30 03:15:05.865404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.865377    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:05.877225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.877187    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4\" (UniqueName: \"kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4\") pod \"f884eb0a-68a1-4fc5-b40d-af25919511b3\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") "
Oct 30 03:15:05.877385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.877255    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca\") pod \"f884eb0a-68a1-4fc5-b40d-af25919511b3\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") "
Oct 30 03:15:05.877385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.877314    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca\") pod \"f884eb0a-68a1-4fc5-b40d-af25919511b3\" (UID: \"f884eb0a-68a1-4fc5-b40d-af25919511b3\") "
Oct 30 03:15:05.877780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.877747    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca" pod "f884eb0a-68a1-4fc5-b40d-af25919511b3" (UID: "f884eb0a-68a1-4fc5-b40d-af25919511b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:05.879498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.879469    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4" pod "f884eb0a-68a1-4fc5-b40d-af25919511b3" (UID: "f884eb0a-68a1-4fc5-b40d-af25919511b3"). InnerVolumeSpecName "kube-api-access-rc6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:05.879600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.879511    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca" pod "f884eb0a-68a1-4fc5-b40d-af25919511b3" (UID: "f884eb0a-68a1-4fc5-b40d-af25919511b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:15:05.978568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.978529    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6q4\" (UniqueName: \"kubernetes.io/projected/f884eb0a-68a1-4fc5-b40d-af25919511b3-kube-api-access-rc6q4\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:05.978568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.978557    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:05.978568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:05.978566    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f884eb0a-68a1-4fc5-b40d-af25919511b3-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:06.529123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:06.529084    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:15:06.529500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:06.529474    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:15:06.530927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:06.530905    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 03:15:06.659770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:06.659738    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"
Oct 30 03:15:06.660233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:06.659770    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt" event={"ID":"f884eb0a-68a1-4fc5-b40d-af25919511b3","Type":"ContainerDied","Data":"2c8e72bf17da3a7f6ab55fb92f44250c254274aa287e0d827750e137f3e43a79"}
Oct 30 03:15:06.660233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:06.659832    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c8e72bf17da3a7f6ab55fb92f44250c254274aa287e0d827750e137f3e43a79"
Oct 30 03:15:16.235113 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:16.235070    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:15:16.690554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:16.690525    4877 scope.go:122] "RemoveContainer" containerID="4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351"
Oct 30 03:15:17.691286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:17.691250    4877 scope.go:122] "RemoveContainer" containerID="913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422"
Oct 30 03:15:17.691718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:17.691628    4877 scope.go:122] "RemoveContainer" containerID="cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b"
Oct 30 03:15:17.744457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:17.744389    4877 generic.go:356] "Generic (PLEG): container finished" podID="3714134a-abab-4e93-a271-ae085eec9bc2" containerID="af23a27098df50612276be7ead6e9c27aefcfcf408aeba2c51ff8763b64bfdaa" exitCode=60
Oct 30 03:15:17.744457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:17.744435    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" event={"ID":"3714134a-abab-4e93-a271-ae085eec9bc2","Type":"ContainerDied","Data":"af23a27098df50612276be7ead6e9c27aefcfcf408aeba2c51ff8763b64bfdaa"}
Oct 30 03:15:17.744697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:17.744493    4877 scope.go:122] "RemoveContainer" containerID="4dbdb211f5ecec1da2c267ee2e933d788319ee2779ec2b8945f325137c15b351"
Oct 30 03:15:17.744921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:17.744903    4877 scope.go:122] "RemoveContainer" containerID="af23a27098df50612276be7ead6e9c27aefcfcf408aeba2c51ff8763b64bfdaa"
Oct 30 03:15:17.745221 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:17.745199    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm_odh-ai-gateway-infra(3714134a-abab-4e93-a271-ae085eec9bc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" podUID="3714134a-abab-4e93-a271-ae085eec9bc2"
Oct 30 03:15:18.753061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.753015    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerStarted","Data":"69bf5e8c055bfad687a04526ec37b1e52fcaaaf5a3ef6187a869830ff56be29b"}
Oct 30 03:15:18.756290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.756263    4877 generic.go:356] "Generic (PLEG): container finished" podID="e78e26d3-3441-431a-8654-94ecd87da441" containerID="86a0ed20c3071284af92ffea165d527a2ff251b4be03fa7d495d9a2e7d8f94e2" exitCode=60
Oct 30 03:15:18.756395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.756333    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" event={"ID":"e78e26d3-3441-431a-8654-94ecd87da441","Type":"ContainerDied","Data":"86a0ed20c3071284af92ffea165d527a2ff251b4be03fa7d495d9a2e7d8f94e2"}
Oct 30 03:15:18.756395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.756386    4877 scope.go:122] "RemoveContainer" containerID="913a4c9380379dde569214e87c7f31c1af766df9114cb5853fb2badc80dce422"
Oct 30 03:15:18.756677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.756660    4877 scope.go:122] "RemoveContainer" containerID="86a0ed20c3071284af92ffea165d527a2ff251b4be03fa7d495d9a2e7d8f94e2"
Oct 30 03:15:18.756902 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:18.756876    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm_odh-ai-gateway-infra(e78e26d3-3441-431a-8654-94ecd87da441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" podUID="e78e26d3-3441-431a-8654-94ecd87da441"
Oct 30 03:15:18.801685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.801656    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"]
Oct 30 03:15:18.962949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:18.962926    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:19.037635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.037603    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w\" (UniqueName: \"kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w\") pod \"3714134a-abab-4e93-a271-ae085eec9bc2\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") "
Oct 30 03:15:19.037832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.037651    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca\") pod \"3714134a-abab-4e93-a271-ae085eec9bc2\" (UID: \"3714134a-abab-4e93-a271-ae085eec9bc2\") "
Oct 30 03:15:19.038090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.038065    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca" pod "3714134a-abab-4e93-a271-ae085eec9bc2" (UID: "3714134a-abab-4e93-a271-ae085eec9bc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:19.039801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.039778    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w" pod "3714134a-abab-4e93-a271-ae085eec9bc2" (UID: "3714134a-abab-4e93-a271-ae085eec9bc2"). InnerVolumeSpecName "kube-api-access-5cl2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:19.139526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.139416    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cl2w\" (UniqueName: \"kubernetes.io/projected/3714134a-abab-4e93-a271-ae085eec9bc2-kube-api-access-5cl2w\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:19.139526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.139465    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3714134a-abab-4e93-a271-ae085eec9bc2-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:19.528378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.528339    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"]
Oct 30 03:15:19.529163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529140    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529160    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529169    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f884eb0a-68a1-4fc5-b40d-af25919511b3" containerName="revoke-keys"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529175    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f884eb0a-68a1-4fc5-b40d-af25919511b3" containerName="revoke-keys"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529199    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529206    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529219    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529224    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529315    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529323    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:19.529580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.529333    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f884eb0a-68a1-4fc5-b40d-af25919511b3" containerName="revoke-keys"
Oct 30 03:15:19.532838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.532817    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.550668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.550630    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"]
Oct 30 03:15:19.557324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.557295    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"openshift-service-ca.crt\""
Oct 30 03:15:19.557497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.557372    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"kube-root-ca.crt\""
Oct 30 03:15:19.557497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.557453    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"default-dockercfg-rlsgn\""
Oct 30 03:15:19.558918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.558897    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2d5b09\"/\"distinct-w2-2d5b09-kserve-self-signed-certs\""
Oct 30 03:15:19.644828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.644790    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.644997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.644940    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.644997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.644967    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.645108 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.645086    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.645152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.645138    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lm9\" (UniqueName: \"kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.645218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.645202    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.746655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.746613    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.746840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.746692    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.746840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.746811    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.746840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.746835    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.747012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.746922    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.747012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.746947    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lm9\" (UniqueName: \"kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.747254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.747231    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.748448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.747595    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.748448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.747610    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.749823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.749795    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.749946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.749923    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.771555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.771404    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lm9\" (UniqueName: \"kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9\") pod \"distinct-w2-2d5b09-kserve-797b7dbd77-m7n75\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") " pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.772694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.772157    4877 generic.go:356] "Generic (PLEG): container finished" podID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerID="69bf5e8c055bfad687a04526ec37b1e52fcaaaf5a3ef6187a869830ff56be29b" exitCode=60
Oct 30 03:15:19.772694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.772281    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerDied","Data":"69bf5e8c055bfad687a04526ec37b1e52fcaaaf5a3ef6187a869830ff56be29b"}
Oct 30 03:15:19.772694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.772351    4877 scope.go:122] "RemoveContainer" containerID="cb44dc5adb22c91f423cf04268d2a1a5e0167696174349d8c673dd54f1c3173b"
Oct 30 03:15:19.772874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.772837    4877 scope.go:122] "RemoveContainer" containerID="69bf5e8c055bfad687a04526ec37b1e52fcaaaf5a3ef6187a869830ff56be29b"
Oct 30 03:15:19.773521 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:19.773492    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888835-7pxsd_odh-ai-gateway-infra(de2cf0f1-23f7-4238-a94d-588d45531df5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5"
Oct 30 03:15:19.774580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.774564    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"
Oct 30 03:15:19.774715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.774586    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm" event={"ID":"3714134a-abab-4e93-a271-ae085eec9bc2","Type":"ContainerDied","Data":"196471e7e22c8ef037beeeead68243a98fe28e371c1dc5a976649bb87b83eecf"}
Oct 30 03:15:19.794455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.794432    4877 scope.go:122] "RemoveContainer" containerID="af23a27098df50612276be7ead6e9c27aefcfcf408aeba2c51ff8763b64bfdaa"
Oct 30 03:15:19.804449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.802983    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"]
Oct 30 03:15:19.839320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.839281    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"]
Oct 30 03:15:19.850536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.850497    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:19.854860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.854832    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"]
Oct 30 03:15:19.867238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:19.867193    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888835-c7tqm"]
Oct 30 03:15:20.032370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.032340    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:20.152806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.152724    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c\" (UniqueName: \"kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c\") pod \"e78e26d3-3441-431a-8654-94ecd87da441\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") "
Oct 30 03:15:20.152806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.152765    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca\") pod \"e78e26d3-3441-431a-8654-94ecd87da441\" (UID: \"e78e26d3-3441-431a-8654-94ecd87da441\") "
Oct 30 03:15:20.153169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.153138    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca" pod "e78e26d3-3441-431a-8654-94ecd87da441" (UID: "e78e26d3-3441-431a-8654-94ecd87da441"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:20.154901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.154879    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c" pod "e78e26d3-3441-431a-8654-94ecd87da441" (UID: "e78e26d3-3441-431a-8654-94ecd87da441"). InnerVolumeSpecName "kube-api-access-mmc8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:20.189105 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:15:20.189078    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e247fa9_29b7_4f01_891a_ea4bde141301.slice/crio-2da9ff843de51af1acc491950ab8705fd525c56b0fedec7e98d6eaab6f534c32 WatchSource:0}: Error finding container 2da9ff843de51af1acc491950ab8705fd525c56b0fedec7e98d6eaab6f534c32: Status 404 returned error can't find the container with id 2da9ff843de51af1acc491950ab8705fd525c56b0fedec7e98d6eaab6f534c32
Oct 30 03:15:20.191092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.191068    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"]
Oct 30 03:15:20.253720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.253691    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmc8c\" (UniqueName: \"kubernetes.io/projected/e78e26d3-3441-431a-8654-94ecd87da441-kube-api-access-mmc8c\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.253819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.253721    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78e26d3-3441-431a-8654-94ecd87da441-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:20.788970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.788928    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" event={"ID":"8e247fa9-29b7-4f01-891a-ea4bde141301","Type":"ContainerStarted","Data":"52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4"}
Oct 30 03:15:20.789369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.788984    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" event={"ID":"8e247fa9-29b7-4f01-891a-ea4bde141301","Type":"ContainerStarted","Data":"2da9ff843de51af1acc491950ab8705fd525c56b0fedec7e98d6eaab6f534c32"}
Oct 30 03:15:20.791733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.791710    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm" event={"ID":"e78e26d3-3441-431a-8654-94ecd87da441","Type":"ContainerDied","Data":"7088a3e5ca977f57ee3288b78408b1a1342fdd19a908c6df0af3473d47809fe2"}
Oct 30 03:15:20.791843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.791764    4877 scope.go:122] "RemoveContainer" containerID="86a0ed20c3071284af92ffea165d527a2ff251b4be03fa7d495d9a2e7d8f94e2"
Oct 30 03:15:20.791843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.791800    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"
Oct 30 03:15:20.851146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.851113    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:20.869016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.868991    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"]
Oct 30 03:15:20.874563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.874533    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888835-gf4zm"]
Oct 30 03:15:20.904292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.904240    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podStartSLOduration=1.904225615 podStartE2EDuration="1.904225615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:20.902689868 +0000 UTC m=+749.331579286" watchObservedRunningTime="2026-10-30 03:15:20.904225615 +0000 UTC m=+749.333115055"
Oct 30 03:15:20.945578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:20.945552    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:21.070412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.070326    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca\") pod \"de2cf0f1-23f7-4238-a94d-588d45531df5\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") "
Oct 30 03:15:21.070581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.070507    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5\" (UniqueName: \"kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5\") pod \"de2cf0f1-23f7-4238-a94d-588d45531df5\" (UID: \"de2cf0f1-23f7-4238-a94d-588d45531df5\") "
Oct 30 03:15:21.070718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.070693    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca" pod "de2cf0f1-23f7-4238-a94d-588d45531df5" (UID: "de2cf0f1-23f7-4238-a94d-588d45531df5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:15:21.070864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.070850    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2cf0f1-23f7-4238-a94d-588d45531df5-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.072403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.072381    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5" pod "de2cf0f1-23f7-4238-a94d-588d45531df5" (UID: "de2cf0f1-23f7-4238-a94d-588d45531df5"). InnerVolumeSpecName "kube-api-access-wlvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:15:21.171682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.171642    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvf5\" (UniqueName: \"kubernetes.io/projected/de2cf0f1-23f7-4238-a94d-588d45531df5-kube-api-access-wlvf5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:15:21.700304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.700272    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" path="/var/lib/kubelet/pods/3714134a-abab-4e93-a271-ae085eec9bc2/volumes"
Oct 30 03:15:21.700748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.700733    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78e26d3-3441-431a-8654-94ecd87da441" path="/var/lib/kubelet/pods/e78e26d3-3441-431a-8654-94ecd87da441/volumes"
Oct 30 03:15:21.800118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.800089    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"
Oct 30 03:15:21.800659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.800131    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd" event={"ID":"de2cf0f1-23f7-4238-a94d-588d45531df5","Type":"ContainerDied","Data":"1d48d35ffe094b90e0848e5270071501d16116137f77227200c9d7f49275696d"}
Oct 30 03:15:21.800659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.800187    4877 scope.go:122] "RemoveContainer" containerID="69bf5e8c055bfad687a04526ec37b1e52fcaaaf5a3ef6187a869830ff56be29b"
Oct 30 03:15:21.853280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.853240    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"]
Oct 30 03:15:21.864404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:21.864374    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888835-7pxsd"]
Oct 30 03:15:23.698407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:23.698370    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" path="/var/lib/kubelet/pods/de2cf0f1-23f7-4238-a94d-588d45531df5/volumes"
Oct 30 03:15:29.861445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:29.860471    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:29.890817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:29.890782    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:30.864995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:30.864963    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:15:40.721770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.721680    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"]
Oct 30 03:15:40.722493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722451    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.722493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722474    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.722493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722487    4877 cpu_manager.go:414] "containerMap: removing container" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:15:40.722493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722495    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722535    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722543    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722553    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722559    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722582    4877 cpu_manager.go:414] "containerMap: removing container" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722588    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:15:40.722745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722735    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:15:40.723062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722748    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.723062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722758    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.723062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722770    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:15:40.723062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722781    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3714134a-abab-4e93-a271-ae085eec9bc2" containerName="cleanup"
Oct 30 03:15:40.723062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.722794    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78e26d3-3441-431a-8654-94ecd87da441" containerName="cleanup"
Oct 30 03:15:40.726694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.726667    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.754922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.754895    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"]
Oct 30 03:15:40.829461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829411    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829475    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829559    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829591    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5s2q\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829621    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829751    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829822    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829859    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.829937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.829921    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.930949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.930915    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.930961    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931009    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931036    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931055    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931069    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5s2q\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931094    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931156    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931230    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931555    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931680    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931695    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931785    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.931867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.931834    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.933544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.933519    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.933851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.933830    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.951033 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.951004    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:40.951324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:40.951303    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5s2q\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:41.040391 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:41.040360    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:41.387547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:41.387514    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"]
Oct 30 03:15:41.389932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:41.389891    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:15:41.390040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:41.389964    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:15:41.390040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:41.390002    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:15:42.082606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:42.082557    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" event={"ID":"9865bd6e-64ed-4c02-9410-a58fd13cab8c","Type":"ContainerStarted","Data":"d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0"}
Oct 30 03:15:42.083299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:42.083264    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" event={"ID":"9865bd6e-64ed-4c02-9410-a58fd13cab8c","Type":"ContainerStarted","Data":"90a1cc4c6bd5d48d4fd311d4af16589d9c216e7c6aee5c37c9d3f539817fd1cc"}
Oct 30 03:15:42.158681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:42.158612    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podStartSLOduration=2.158587994 podStartE2EDuration="2.158587994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:42.135710684 +0000 UTC m=+770.564600104" watchObservedRunningTime="2026-10-30 03:15:42.158587994 +0000 UTC m=+770.587477423"
Oct 30 03:15:43.040648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:43.040581    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:43.042776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:43.042741    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:15:43.043026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:43.042993    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 03:15:44.041398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:44.041360    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:15:44.041836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:44.041413    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 03:15:45.040693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:45.040660    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:15:45.040857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:45.040717    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 03:15:46.041738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:46.041703    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:15:46.042123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:46.041772    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 03:15:46.233757 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:15:46.233721    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:15:47.040834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:47.040802    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:15:47.041020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:47.040865    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 03:15:48.041213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:48.041181    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 03:15:48.041721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:48.041235    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 03:15:49.045188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:49.045157    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:49.045715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:49.045652    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:15:49.046442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:15:49.046407    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:16:16.228475 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:16:16.228409    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:16:46.178740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.178709    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"]
Oct 30 03:16:46.179285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.179270    4877 cpu_manager.go:414] "containerMap: removing container" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:16:46.179328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.179286    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:16:46.179376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.179367    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2cf0f1-23f7-4238-a94d-588d45531df5" containerName="cleanup"
Oct 30 03:16:46.182860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.182842    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.209825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.209796    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3c874f\"/\"distinct-w3-3c874f-kserve-self-signed-certs\""
Oct 30 03:16:46.214203 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.214171    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"]
Oct 30 03:16:46.237802 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:16:46.237770    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:16:46.269942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.269912    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4b74\" (UniqueName: \"kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.270069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.269955    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.270111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.270067    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.270167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.270155    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.270230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.270211    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.270270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.270243    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.370892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.370860    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.370973    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371007    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371029    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371074    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4b74\" (UniqueName: \"kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371116    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371341    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371379    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.371670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.371441    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.379234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.378747    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.379234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.379023    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.416690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.416653    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4b74\" (UniqueName: \"kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74\") pod \"distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") " pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.496682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.496598    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:46.803843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:46.803809    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"]
Oct 30 03:16:46.804164 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:16:46.804142    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod492247f6_5d6b_4060_b7db_a5cec11a4ea5.slice/crio-f16e6eb768e8766ba4e85bd391d6f2d944e66e1657c210c2744d7525768ca253 WatchSource:0}: Error finding container f16e6eb768e8766ba4e85bd391d6f2d944e66e1657c210c2744d7525768ca253: Status 404 returned error can't find the container with id f16e6eb768e8766ba4e85bd391d6f2d944e66e1657c210c2744d7525768ca253
Oct 30 03:16:47.588736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:47.588699    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" event={"ID":"492247f6-5d6b-4060-b7db-a5cec11a4ea5","Type":"ContainerStarted","Data":"a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2"}
Oct 30 03:16:47.589309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:47.589282    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" event={"ID":"492247f6-5d6b-4060-b7db-a5cec11a4ea5","Type":"ContainerStarted","Data":"f16e6eb768e8766ba4e85bd391d6f2d944e66e1657c210c2744d7525768ca253"}
Oct 30 03:16:47.629440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:47.629353    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podStartSLOduration=1.629336059 podStartE2EDuration="1.629336059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:47.628185066 +0000 UTC m=+836.057074483" watchObservedRunningTime="2026-10-30 03:16:47.629336059 +0000 UTC m=+836.058225488"
Oct 30 03:16:48.497147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:48.497109    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:56.497835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:56.497801    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:56.510815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:56.510786    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:16:56.673604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:16:56.673573    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:17:16.229307 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:17:16.229223    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:17:46.228741 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:17:46.228704    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:17:51.932823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:17:51.932796    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:17:51.933336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:17:51.933073    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:18:16.228643 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:18:16.228604    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:18:18.456005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:18.455973    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"]
Oct 30 03:18:18.456398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:18.456184    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy" containerID="cri-o://6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5" gracePeriod=30
Oct 30 03:18:23.737219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.737192    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:18:23.882123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882096    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882230    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882306    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882329    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882350    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882376    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882411    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882484    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882512    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert\") pod \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\" (UID: \"cce76e7f-a3b1-4099-bc03-1c2d5e33d935\") "
Oct 30 03:18:23.882754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882730    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:23.882817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882745    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:23.882817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.882762    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:23.883035 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.883008    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:23.883123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.883032    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.883123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.883057    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.883123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.883056    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:23.883123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.883071    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.884447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.884403    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "kube-api-access-fkklm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:23.884856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.884826    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:18:23.884969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.884924    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:23.884969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.884930    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy" pod "cce76e7f-a3b1-4099-bc03-1c2d5e33d935" (UID: "cce76e7f-a3b1-4099-bc03-1c2d5e33d935"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:23.984294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.984204    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.984294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.984233    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.984294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.984241    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.984294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.984250    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.984294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.984258    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkklm\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-kube-api-access-fkklm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:23.984294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:23.984269    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce76e7f-a3b1-4099-bc03-1c2d5e33d935-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:24.282049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.282013    4877 generic.go:356] "Generic (PLEG): container finished" podID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerID="6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5" exitCode=0
Oct 30 03:18:24.282251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.282067    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" event={"ID":"cce76e7f-a3b1-4099-bc03-1c2d5e33d935","Type":"ContainerDied","Data":"6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5"}
Oct 30 03:18:24.282251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.282068    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"
Oct 30 03:18:24.282251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.282111    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96" event={"ID":"cce76e7f-a3b1-4099-bc03-1c2d5e33d935","Type":"ContainerDied","Data":"190604e77fa7d6cdd36fbb20d4a5de6f687d748a8acb6ec7c3c0c12922596196"}
Oct 30 03:18:24.282251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.282131    4877 scope.go:122] "RemoveContainer" containerID="6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5"
Oct 30 03:18:24.295863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.295838    4877 scope.go:122] "RemoveContainer" containerID="6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5"
Oct 30 03:18:24.296279 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:18:24.296255    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5\": container with ID starting with 6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5 not found: ID does not exist" containerID="6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5"
Oct 30 03:18:24.296411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.296289    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5"} err="failed to get container status \"6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5\": rpc error: code = NotFound desc = could not find container \"6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5\": container with ID starting with 6ba0fbe625f0fca68e899a477c166ce130c177677778ce4c3916952a5bc782e5 not found: ID does not exist"
Oct 30 03:18:24.334395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.334367    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"]
Oct 30 03:18:24.346307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:24.346281    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-8cf96"]
Oct 30 03:18:25.698562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:25.698529    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" path="/var/lib/kubelet/pods/cce76e7f-a3b1-4099-bc03-1c2d5e33d935/volumes"
Oct 30 03:18:42.454414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.454287    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"]
Oct 30 03:18:42.455172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.455146    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy"
Oct 30 03:18:42.455267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.455172    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy"
Oct 30 03:18:42.455383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.455365    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce76e7f-a3b1-4099-bc03-1c2d5e33d935" containerName="istio-proxy"
Oct 30 03:18:42.459714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.459692    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.474270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.474248    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b132360f-openshift-default-dockercfg-blxjp\""
Oct 30 03:18:42.490353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.490327    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"]
Oct 30 03:18:42.575969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.575936    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.575972    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576076    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576107    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576131    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576262    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576301    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576453    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.576514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.576481    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwnv\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.681077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.681039    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.691786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.691752    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwnv\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.692038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.692015    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.692237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.692210    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.692393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.692311    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.692393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.692358    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.692565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.692395    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.698485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.689863    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.700505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.700473    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.700646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.700528    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.701634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.701605    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.702308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.702279    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.703714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.703690    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.703973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.703955    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.704635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.704585    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.705331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.705302    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.719377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.719349    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.720228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.720202    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwnv\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv\") pod \"e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") " pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:42.775123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:42.775087    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:43.098664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.098629    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"]
Oct 30 03:18:43.099626 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:18:43.099600    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e16d76a_e3d5_4ca1_8f4d_dae69b24f55f.slice/crio-17603b5150b0ab5b9dbb57b8ad00ec2ba982cd900694263dd72b85ff45a2db3f WatchSource:0}: Error finding container 17603b5150b0ab5b9dbb57b8ad00ec2ba982cd900694263dd72b85ff45a2db3f: Status 404 returned error can't find the container with id 17603b5150b0ab5b9dbb57b8ad00ec2ba982cd900694263dd72b85ff45a2db3f
Oct 30 03:18:43.101977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.101945    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:18:43.102055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.102008    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:18:43.102055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.102040    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:18:43.428674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.428564    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" event={"ID":"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f","Type":"ContainerStarted","Data":"756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"}
Oct 30 03:18:43.428674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.428643    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" event={"ID":"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f","Type":"ContainerStarted","Data":"17603b5150b0ab5b9dbb57b8ad00ec2ba982cd900694263dd72b85ff45a2db3f"}
Oct 30 03:18:43.474243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.474184    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podStartSLOduration=1.474165716 podStartE2EDuration="1.474165716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:43.469355325 +0000 UTC m=+951.898244777" watchObservedRunningTime="2026-10-30 03:18:43.474165716 +0000 UTC m=+951.903055136"
Oct 30 03:18:43.775165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:43.775137    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:44.781873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:44.781843    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:45.446490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:45.446455    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:45.447315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:45.447290    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:18:46.230767 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:18:46.230726    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:18:47.744883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.744834    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"]
Oct 30 03:18:47.750006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.749985    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:47.767077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.767053    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"]
Oct 30 03:18:47.773831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.773789    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b132360f\""
Oct 30 03:18:47.773960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.773947    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b132360f-dockercfg-xpjjr\""
Oct 30 03:18:47.878209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.878164    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrj4\" (UniqueName: \"kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-jsx99\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:47.892487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.885637    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-jsx99\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:47.994017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.993980    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrj4\" (UniqueName: \"kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-jsx99\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:47.994236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.994052    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-jsx99\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:47.995055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:47.994986    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-jsx99\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:48.013354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.013324    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrj4\" (UniqueName: \"kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-jsx99\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:48.060201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.060168    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"]
Oct 30 03:18:48.064080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.064063    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.066328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.066305    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:48.085676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.085591    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"]
Oct 30 03:18:48.100465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.100010    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.100465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.100066    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.100465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.100091    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gldj9\" (UniqueName: \"kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.162934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.162902    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:18:48.163095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.163032    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:18:48.200963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.200930    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.201104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.200987    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.201104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.201015    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gldj9\" (UniqueName: \"kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.201761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.201738    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.203808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.203761    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.219783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.219760    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gldj9\" (UniqueName: \"kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9\") pod \"maas-api-revoke-sub-28a3eaf3-46ccg\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.382132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.382100    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"]
Oct 30 03:18:48.382649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.382619    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:48.383561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:18:48.383535    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d90b18c_3b2e_4058_9d17_a0f33633c95b.slice/crio-62fea93f30cbab51b62fab3544f3bdb7038247f4831adc892269e03d41a6c10b WatchSource:0}: Error finding container 62fea93f30cbab51b62fab3544f3bdb7038247f4831adc892269e03d41a6c10b: Status 404 returned error can't find the container with id 62fea93f30cbab51b62fab3544f3bdb7038247f4831adc892269e03d41a6c10b
Oct 30 03:18:48.490265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.490217    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" event={"ID":"5d90b18c-3b2e-4058-9d17-a0f33633c95b","Type":"ContainerStarted","Data":"62fea93f30cbab51b62fab3544f3bdb7038247f4831adc892269e03d41a6c10b"}
Oct 30 03:18:48.490413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.490327    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:18:48.530065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.530012    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" podStartSLOduration=1.5299971700000001 podStartE2EDuration="1.52999717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:48.528845226 +0000 UTC m=+956.957734652" watchObservedRunningTime="2026-10-30 03:18:48.52999717 +0000 UTC m=+956.958886588"
Oct 30 03:18:48.694930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:48.694900    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"]
Oct 30 03:18:48.695393 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:18:48.695367    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd6ca6d_ab78_4c51_96f2_82c38b435c17.slice/crio-15dd22705e4871878a0ae2b93a5fda3a8f90d289ff7942ff6a07a10d5c48d0bb WatchSource:0}: Error finding container 15dd22705e4871878a0ae2b93a5fda3a8f90d289ff7942ff6a07a10d5c48d0bb: Status 404 returned error can't find the container with id 15dd22705e4871878a0ae2b93a5fda3a8f90d289ff7942ff6a07a10d5c48d0bb
Oct 30 03:18:49.499834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:49.499729    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" event={"ID":"5d90b18c-3b2e-4058-9d17-a0f33633c95b","Type":"ContainerStarted","Data":"417725000f9a430c42a4f693e9ed42d8d5de5e44af35b2437f64804960bcb815"}
Oct 30 03:18:49.501942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:49.501911    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg" event={"ID":"6dd6ca6d-ab78-4c51-96f2-82c38b435c17","Type":"ContainerStarted","Data":"95303021abfabe005a4963b2cd5beb688e46a67cd3ada0937f4aa5d842ae297e"}
Oct 30 03:18:49.502075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:49.501959    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg" event={"ID":"6dd6ca6d-ab78-4c51-96f2-82c38b435c17","Type":"ContainerStarted","Data":"15dd22705e4871878a0ae2b93a5fda3a8f90d289ff7942ff6a07a10d5c48d0bb"}
Oct 30 03:18:49.545698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:49.545641    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg" podStartSLOduration=1.545627932 podStartE2EDuration="1.545627932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:49.543232943 +0000 UTC m=+957.972122361" watchObservedRunningTime="2026-10-30 03:18:49.545627932 +0000 UTC m=+957.974517350"
Oct 30 03:18:50.511967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:50.511928    4877 generic.go:356] "Generic (PLEG): container finished" podID="6dd6ca6d-ab78-4c51-96f2-82c38b435c17" containerID="95303021abfabe005a4963b2cd5beb688e46a67cd3ada0937f4aa5d842ae297e" exitCode=0
Oct 30 03:18:50.512492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:50.512024    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg" event={"ID":"6dd6ca6d-ab78-4c51-96f2-82c38b435c17","Type":"ContainerDied","Data":"95303021abfabe005a4963b2cd5beb688e46a67cd3ada0937f4aa5d842ae297e"}
Oct 30 03:18:51.768603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.768575    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:51.865669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.865566    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca\") pod \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") "
Oct 30 03:18:51.865669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.865658    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9\" (UniqueName: \"kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9\") pod \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") "
Oct 30 03:18:51.865954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.865716    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca\") pod \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\" (UID: \"6dd6ca6d-ab78-4c51-96f2-82c38b435c17\") "
Oct 30 03:18:51.866044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.865994    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca" pod "6dd6ca6d-ab78-4c51-96f2-82c38b435c17" (UID: "6dd6ca6d-ab78-4c51-96f2-82c38b435c17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:51.866207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.866189    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:51.868075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.868048    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca" pod "6dd6ca6d-ab78-4c51-96f2-82c38b435c17" (UID: "6dd6ca6d-ab78-4c51-96f2-82c38b435c17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:51.868293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.868269    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9" pod "6dd6ca6d-ab78-4c51-96f2-82c38b435c17" (UID: "6dd6ca6d-ab78-4c51-96f2-82c38b435c17"). InnerVolumeSpecName "kube-api-access-gldj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:51.949225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.949185    4877 scope.go:122] "RemoveContainer" containerID="23859c9eed461f089dbb783cd1a9634f8eb31758afe1325920f184f004633a84"
Oct 30 03:18:51.960864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.960836    4877 scope.go:122] "RemoveContainer" containerID="e7e521ebb13c57679a2d531434ac44afd0472f78cea20f5124e25087999b3d4c"
Oct 30 03:18:51.969493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.968033    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gldj9\" (UniqueName: \"kubernetes.io/projected/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-kube-api-access-gldj9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:51.969493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.968065    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd6ca6d-ab78-4c51-96f2-82c38b435c17-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:18:51.973711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.973692    4877 scope.go:122] "RemoveContainer" containerID="e56e6469b0134e02be0858b90be3ebcdf5b25a625ac6269a88db714f8ac464d4"
Oct 30 03:18:51.985345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:51.985318    4877 scope.go:122] "RemoveContainer" containerID="87b26226e005de82c805d9cfbeddab226aa3aaba078ecfe84c908a97bcde8ab0"
Oct 30 03:18:52.580327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:52.580287    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"
Oct 30 03:18:52.580952 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:52.580286    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg" event={"ID":"6dd6ca6d-ab78-4c51-96f2-82c38b435c17","Type":"ContainerDied","Data":"15dd22705e4871878a0ae2b93a5fda3a8f90d289ff7942ff6a07a10d5c48d0bb"}
Oct 30 03:18:52.581059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:18:52.580989    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15dd22705e4871878a0ae2b93a5fda3a8f90d289ff7942ff6a07a10d5c48d0bb"
Oct 30 03:19:00.515615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:00.515584    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:19:06.623624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.623585    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"]
Oct 30 03:19:06.625096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.625065    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6dd6ca6d-ab78-4c51-96f2-82c38b435c17" containerName="revoke-keys"
Oct 30 03:19:06.625096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.625094    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd6ca6d-ab78-4c51-96f2-82c38b435c17" containerName="revoke-keys"
Oct 30 03:19:06.625349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.625287    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd6ca6d-ab78-4c51-96f2-82c38b435c17" containerName="revoke-keys"
Oct 30 03:19:06.631518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.631497    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.646825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.646804    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f247d1-openshift-default-dockercfg-nbqkl\""
Oct 30 03:19:06.664933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.664902    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"]
Oct 30 03:19:06.781371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.781323    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.787179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.787149    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.793469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.787476    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.793781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.793755    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.794070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.794049    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkmg\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.794240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.794226    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.794462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.794445    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.794579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.794564    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.794680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.794668    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895396    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895448    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895482    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895512    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895547    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkmg\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895610    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895703    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895724    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.895798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895755    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.896121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.895988    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.896257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.896222    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.896381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.896353    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.896381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.896284    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.896593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.896357    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.898330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.898307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.898458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.898438    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.915475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.915450    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.915613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.915589    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkmg\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg\") pod \"e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") " pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:06.946457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:06.946412    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:07.273610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.273575    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"]
Oct 30 03:19:07.273977 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:19:07.273955    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5ad99e_329e_4a1e_acfb_da8960421f17.slice/crio-300513df7277ea294b6bfbc99b1504b891e67d03f05edcf982b0f279832c8616 WatchSource:0}: Error finding container 300513df7277ea294b6bfbc99b1504b891e67d03f05edcf982b0f279832c8616: Status 404 returned error can't find the container with id 300513df7277ea294b6bfbc99b1504b891e67d03f05edcf982b0f279832c8616
Oct 30 03:19:07.276308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.276276    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:07.276376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.276340    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:07.276376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.276367    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:07.702316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.702223    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" event={"ID":"ca5ad99e-329e-4a1e-acfb-da8960421f17","Type":"ContainerStarted","Data":"a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"}
Oct 30 03:19:07.702316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.702271    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" event={"ID":"ca5ad99e-329e-4a1e-acfb-da8960421f17","Type":"ContainerStarted","Data":"300513df7277ea294b6bfbc99b1504b891e67d03f05edcf982b0f279832c8616"}
Oct 30 03:19:07.767901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.767852    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podStartSLOduration=1.7678379739999999 podStartE2EDuration="1.767837974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:07.755705166 +0000 UTC m=+976.184594607" watchObservedRunningTime="2026-10-30 03:19:07.767837974 +0000 UTC m=+976.196727456"
Oct 30 03:19:07.947116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:07.947071    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:08.955401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:08.955349    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:09.716766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:09.716732    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:09.717715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:09.717686    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:12.199909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.190503    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"]
Oct 30 03:19:12.199909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.195775    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.211320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.211188    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f247d1\""
Oct 30 03:19:12.248186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.248157    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f247d1-dockercfg-fgcln\""
Oct 30 03:19:12.304448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.289567    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krwd\" (UniqueName: \"kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd\") pod \"payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") " pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.304879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.304836    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") " pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.324734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.324673    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"]
Oct 30 03:19:12.417446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.416988    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krwd\" (UniqueName: \"kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd\") pod \"payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") " pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.417446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.417141    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") " pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.420445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.418062    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") " pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.441116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.441090    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krwd\" (UniqueName: \"kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd\") pod \"payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") " pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.569085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.569048    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:12.909170 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:19:12.909140    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff34af_416c_482c_a6ef_9fe68198d4a5.slice/crio-50217bfa832144568a27bc6f97d0fbdc870e74aa6b16642b30193d3fedd9ecf8 WatchSource:0}: Error finding container 50217bfa832144568a27bc6f97d0fbdc870e74aa6b16642b30193d3fedd9ecf8: Status 404 returned error can't find the container with id 50217bfa832144568a27bc6f97d0fbdc870e74aa6b16642b30193d3fedd9ecf8
Oct 30 03:19:12.911281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:12.911254    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"]
Oct 30 03:19:13.750843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:13.750806    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq" event={"ID":"0eff34af-416c-482c-a6ef-9fe68198d4a5","Type":"ContainerStarted","Data":"60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874"}
Oct 30 03:19:13.751286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:13.750861    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq" event={"ID":"0eff34af-416c-482c-a6ef-9fe68198d4a5","Type":"ContainerStarted","Data":"50217bfa832144568a27bc6f97d0fbdc870e74aa6b16642b30193d3fedd9ecf8"}
Oct 30 03:19:13.751286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:13.751053    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:13.797449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:13.793695    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq" podStartSLOduration=1.79367374 podStartE2EDuration="1.79367374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:13.79189684 +0000 UTC m=+982.220786258" watchObservedRunningTime="2026-10-30 03:19:13.79367374 +0000 UTC m=+982.222563159"
Oct 30 03:19:16.229041 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:19:16.228953    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:19:18.640672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.640637    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"]
Oct 30 03:19:18.646233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.646210    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.710384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.710344    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"]
Oct 30 03:19:18.810437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810397    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.810633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810581    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.810757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810723    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c68\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.810887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810797    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.810887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810836    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.811021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810921    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.811021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810966    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.811021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.810994    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.811185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.811061    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.913758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913659    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.913758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913757    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c68\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913799    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913831    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913873    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913904    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913936    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.913989    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.914077    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.914161    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.914371    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.914958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.914929    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.915123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.915097    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.915535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.915397    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.918170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.918142    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.927002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.926969    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.932963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.932937    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c68\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.932963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.932956    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hcwvj\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:18.960727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:18.960698    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:19.306362 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:19:19.306325    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fea061_7b0d_4148_84cf_e82942c93082.slice/crio-c0e0ed253213a99b14ca138c011c8d40e18b5dd1cbb3151ba64cafbf7f744f78 WatchSource:0}: Error finding container c0e0ed253213a99b14ca138c011c8d40e18b5dd1cbb3151ba64cafbf7f744f78: Status 404 returned error can't find the container with id c0e0ed253213a99b14ca138c011c8d40e18b5dd1cbb3151ba64cafbf7f744f78
Oct 30 03:19:19.307177 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.307149    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"]
Oct 30 03:19:19.309320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.309279    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:19.309752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.309367    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:19.309752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.309417    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:19.805826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.805783    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" event={"ID":"79fea061-7b0d-4148-84cf-e82942c93082","Type":"ContainerStarted","Data":"b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234"}
Oct 30 03:19:19.806343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.805854    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" event={"ID":"79fea061-7b0d-4148-84cf-e82942c93082","Type":"ContainerStarted","Data":"c0e0ed253213a99b14ca138c011c8d40e18b5dd1cbb3151ba64cafbf7f744f78"}
Oct 30 03:19:19.808235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.808205    4877 generic.go:356] "Generic (PLEG): container finished" podID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerID="756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e" exitCode=0
Oct 30 03:19:19.808376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.808276    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" event={"ID":"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f","Type":"ContainerDied","Data":"756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"}
Oct 30 03:19:19.808667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.808652    4877 scope.go:122] "RemoveContainer" containerID="756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"
Oct 30 03:19:19.810057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.810030    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:19.810165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.810114    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:19.810165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.810145    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:19.855823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.855755    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podStartSLOduration=1.8557365799999999 podStartE2EDuration="1.85573658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:19.85152114 +0000 UTC m=+988.280410578" watchObservedRunningTime="2026-10-30 03:19:19.85573658 +0000 UTC m=+988.284626001"
Oct 30 03:19:19.961791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:19.961757    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:20.038027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:20.037991    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"]
Oct 30 03:19:20.041958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:20.041927    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10201b1-vqxgv"]
Oct 30 03:19:20.850539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:20.850500    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" event={"ID":"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f","Type":"ContainerStarted","Data":"228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59"}
Oct 30 03:19:20.962994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:20.962962    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:20.963179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:20.963019    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:21.739176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:21.739135    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ceccd54-3780-4001-a332-bb05dd7aaa91" path="/var/lib/kubelet/pods/2ceccd54-3780-4001-a332-bb05dd7aaa91/volumes"
Oct 30 03:19:21.776194 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:21.776163    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:19:21.786448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:21.776469    4877 patch_prober.go:29] interesting pod/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 03:19:21.786448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:21.776523    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 03:19:21.961877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:21.961826    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:21.962292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:21.961897    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:22.775799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:22.775765    4877 patch_prober.go:29] interesting pod/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 03:19:22.775987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:22.775827    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 03:19:22.961895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:22.961852    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:22.962388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:22.961912    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:23.776148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:23.775495    4877 patch_prober.go:29] interesting pod/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 03:19:23.776148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:23.775563    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 03:19:23.962058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:23.962020    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:23.962650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:23.962616    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:24.151400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.151304    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"]
Oct 30 03:19:24.159118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.159069    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.159852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.158871    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"]
Oct 30 03:19:24.202442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.200181    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-585938\""
Oct 30 03:19:24.202442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.200329    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-585938-dockercfg-b7dqr\""
Oct 30 03:19:24.254510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.254466    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.254824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.254801    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdds\" (UniqueName: \"kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds\") pod \"payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.319446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.315544    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"]
Oct 30 03:19:24.355666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.355629    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"]
Oct 30 03:19:24.359656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.359617    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.395158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.395118    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.395494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.395471    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdds\" (UniqueName: \"kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds\") pod \"payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.408701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.396393    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.453775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.453743    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdds\" (UniqueName: \"kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds\") pod \"payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.497173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.497128    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") " pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.497566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.497545    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw7w7\" (UniqueName: \"kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7\") pod \"payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") " pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.600517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.600465    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw7w7\" (UniqueName: \"kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7\") pod \"payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") " pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.601324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.601294    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") " pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.602770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.602751    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") " pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.619756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.619725    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:24.637330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.637288    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw7w7\" (UniqueName: \"kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7\") pod \"payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") " pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.754489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.751686    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:24.772055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.772034    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:19:24.850651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.849575    4877 patch_prober.go:29] interesting pod/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 03:19:24.850651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.849624    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 03:19:24.963813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.963545    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:24.963813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:24.963605    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:25.072284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.072253    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"]
Oct 30 03:19:25.449783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.448391    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"]
Oct 30 03:19:25.453068 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:19:25.453024    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc03823_8f9c_4638_b09c_cea0777ceaea.slice/crio-f8e4d58559264fc084ba8784b5a80dadd6d43c8b8332f690ea400201feef1618 WatchSource:0}: Error finding container f8e4d58559264fc084ba8784b5a80dadd6d43c8b8332f690ea400201feef1618: Status 404 returned error can't find the container with id f8e4d58559264fc084ba8784b5a80dadd6d43c8b8332f690ea400201feef1618
Oct 30 03:19:25.776401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.776366    4877 patch_prober.go:29] interesting pod/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 03:19:25.776621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.776451    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 03:19:25.961322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.961283    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:25.961515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.961352    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:25.986189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.986156    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs" event={"ID":"bdc03823-8f9c-4638-b09c-cea0777ceaea","Type":"ContainerStarted","Data":"cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3"}
Oct 30 03:19:25.986722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.986221    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs" event={"ID":"bdc03823-8f9c-4638-b09c-cea0777ceaea","Type":"ContainerStarted","Data":"f8e4d58559264fc084ba8784b5a80dadd6d43c8b8332f690ea400201feef1618"}
Oct 30 03:19:25.987141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.987119    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:25.989186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.989158    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx" event={"ID":"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c","Type":"ContainerStarted","Data":"cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f"}
Oct 30 03:19:25.989302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.989213    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx" event={"ID":"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c","Type":"ContainerStarted","Data":"12ec2019e73a26378923f46aa5e5d7f45cfe137217495a352973753270636493"}
Oct 30 03:19:25.989371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:25.989316    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:26.029028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.028917    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs" podStartSLOduration=2.028901435 podStartE2EDuration="2.028901435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:26.023515653 +0000 UTC m=+994.452405072" watchObservedRunningTime="2026-10-30 03:19:26.028901435 +0000 UTC m=+994.457790854"
Oct 30 03:19:26.061380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.061007    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx" podStartSLOduration=2.060989355 podStartE2EDuration="2.060989355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:26.060378644 +0000 UTC m=+994.489268111" watchObservedRunningTime="2026-10-30 03:19:26.060989355 +0000 UTC m=+994.489878774"
Oct 30 03:19:26.797921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.797883    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:19:26.798655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.798631    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:19:26.800612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.800590    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:19:26.961192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.961155    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 03:19:26.961385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:26.961212    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 03:19:28.387803 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:28.387570    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:28.388308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:28.388218    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:28.389171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:28.389152    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:19:29.027151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:29.027120    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"]
Oct 30 03:19:29.038433 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:29.038391    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb7a3959-zncq9"]
Oct 30 03:19:29.699921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:29.699891    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc61acff-9d79-46f0-92e6-652202315377" path="/var/lib/kubelet/pods/fc61acff-9d79-46f0-92e6-652202315377/volumes"
Oct 30 03:19:33.741256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.741220    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"]
Oct 30 03:19:33.746881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.746847    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.771853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.771818    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"]
Oct 30 03:19:33.860048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860013    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860107    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860179    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860265    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6cx\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860313    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860359    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860409    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860532    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.860587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.860577    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961373    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6cx\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961419    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961483    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961530    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961590    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961685    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961737    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.961910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961802    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.962066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.961947    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.962066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.962025    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.962339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.962314    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.962459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.962343    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.962459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.962343    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.963796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.963774    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.964178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.964159    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.980117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.980086    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:33.980920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:33.980897    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6cx\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r9z2t\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:34.061102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:34.061072    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:34.388468 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:19:34.388406    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbe6556_4143_44df_b4a0_232d5971626f.slice/crio-ed200e1485f0169aff5c2f0b8938f1d4c39b249ece47a487ed4ff1bfe820ca83 WatchSource:0}: Error finding container ed200e1485f0169aff5c2f0b8938f1d4c39b249ece47a487ed4ff1bfe820ca83: Status 404 returned error can't find the container with id ed200e1485f0169aff5c2f0b8938f1d4c39b249ece47a487ed4ff1bfe820ca83
Oct 30 03:19:34.389363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:34.389331    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"]
Oct 30 03:19:34.391005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:34.390973    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:34.391130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:34.391036    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:34.391130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:34.391068    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:35.069876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:35.069839    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" event={"ID":"ebbe6556-4143-44df-b4a0-232d5971626f","Type":"ContainerStarted","Data":"42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2"}
Oct 30 03:19:35.070266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:35.069921    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" event={"ID":"ebbe6556-4143-44df-b4a0-232d5971626f","Type":"ContainerStarted","Data":"ed200e1485f0169aff5c2f0b8938f1d4c39b249ece47a487ed4ff1bfe820ca83"}
Oct 30 03:19:35.149715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:35.149657    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podStartSLOduration=2.149644093 podStartE2EDuration="2.149644093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:35.134983246 +0000 UTC m=+1003.563872682" watchObservedRunningTime="2026-10-30 03:19:35.149644093 +0000 UTC m=+1003.578533511"
Oct 30 03:19:36.025463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:36.025413    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"]
Oct 30 03:19:36.037069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:36.037034    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206d9152-86g42"]
Oct 30 03:19:36.062170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:36.062135    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:36.064145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:36.064119    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:36.064244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:36.064169    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:37.025447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:37.018619    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:19:37.066823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:37.066779    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:37.067010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:37.066841    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:37.711039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:37.711007    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e" path="/var/lib/kubelet/pods/f1e6dd3c-f13b-4935-aa2f-9ea53a8a423e/volumes"
Oct 30 03:19:38.011565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:38.011465    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:19:38.061478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:38.061445    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:38.061877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:38.061495    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:39.061686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:39.061644    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:39.062191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:39.061705    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:40.061857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:40.061815    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:40.062492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:40.062455    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:41.061635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:41.061593    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:41.061846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:41.061653    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:42.062055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:42.062006    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 03:19:42.062625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:42.062115    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 03:19:43.396448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:43.392934    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:43.396448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:43.394260    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:43.411450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:43.405771    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:19:44.029478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:44.029446    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"]
Oct 30 03:19:44.032900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:44.032869    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc6bb64e-rm84g"]
Oct 30 03:19:45.702739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:45.702701    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd99b12e-3fe4-445d-b891-6a69f114018a" path="/var/lib/kubelet/pods/cd99b12e-3fe4-445d-b891-6a69f114018a/volumes"
Oct 30 03:19:46.229551 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:19:46.229512    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:19:48.260942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:48.260909    4877 generic.go:356] "Generic (PLEG): container finished" podID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerID="a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b" exitCode=0
Oct 30 03:19:48.261394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:48.260969    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" event={"ID":"ca5ad99e-329e-4a1e-acfb-da8960421f17","Type":"ContainerDied","Data":"a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"}
Oct 30 03:19:48.261593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:48.261571    4877 scope.go:122] "RemoveContainer" containerID="a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"
Oct 30 03:19:48.263190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:48.263154    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:48.263312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:48.263233    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:48.263312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:48.263267    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:19:49.275572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:49.274843    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" event={"ID":"ca5ad99e-329e-4a1e-acfb-da8960421f17","Type":"ContainerStarted","Data":"3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181"}
Oct 30 03:19:49.947544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:49.947503    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:49.947742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:49.947699    4877 patch_prober.go:29] interesting pod/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 03:19:49.947795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:49.947755    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 03:19:50.947637 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:50.947595    4877 patch_prober.go:29] interesting pod/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 03:19:50.948038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:50.947691    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 03:19:51.039528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:51.039498    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"]
Oct 30 03:19:51.051766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:51.051711    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf00eb2-rxv4f"]
Oct 30 03:19:51.713522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:51.713484    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85f271e-f98f-4c06-96fe-efb267c92169" path="/var/lib/kubelet/pods/c85f271e-f98f-4c06-96fe-efb267c92169/volumes"
Oct 30 03:19:51.947165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:51.947128    4877 patch_prober.go:29] interesting pod/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 03:19:51.947345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:51.947189    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 03:19:52.785448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.779043    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"]
Oct 30 03:19:52.805569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.805541    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:52.811058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.811028    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"]
Oct 30 03:19:52.833481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.833411    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3e4daf\""
Oct 30 03:19:52.833481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.833449    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3e4daf-dockercfg-jpbbt\""
Oct 30 03:19:52.839045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.839022    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"]
Oct 30 03:19:52.843260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.843241    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:52.871137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.871111    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"]
Oct 30 03:19:52.997451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.964112    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:52.997925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.997899    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vfd\" (UniqueName: \"kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd\") pod \"payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:53.004450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.997798    4877 patch_prober.go:29] interesting pod/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 03:19:53.004450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:52.999618    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 03:19:53.004450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.000038    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.004450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.000076    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkh4\" (UniqueName: \"kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4\") pod \"payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.105477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.105353    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:53.105741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.105718    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vfd\" (UniqueName: \"kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd\") pod \"payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:53.105883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.105866    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.105997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.105983    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkh4\" (UniqueName: \"kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4\") pod \"payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.106727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.106700    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:53.108006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.107980    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.125862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.125831    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkh4\" (UniqueName: \"kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4\") pod \"payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.125963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.125900    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vfd\" (UniqueName: \"kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd\") pod \"payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:53.166009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.165976    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:53.423024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.422938    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:53.523204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.523160    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"]
Oct 30 03:19:53.755387 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:19:53.755352    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd7f9e2_7f68_4d1d_8f33_ca65ae8dc619.slice/crio-6a69b7d180e346a1180601d5b0a5ba2e5daceea52d0407f64d31a197da720ed5 WatchSource:0}: Error finding container 6a69b7d180e346a1180601d5b0a5ba2e5daceea52d0407f64d31a197da720ed5: Status 404 returned error can't find the container with id 6a69b7d180e346a1180601d5b0a5ba2e5daceea52d0407f64d31a197da720ed5
Oct 30 03:19:53.757520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.757462    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"]
Oct 30 03:19:53.946854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.946806    4877 patch_prober.go:29] interesting pod/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 03:19:53.947257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:53.946870    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 03:19:54.363733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.363698    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p" event={"ID":"8c15638e-ea25-4fad-932c-5a614d44f05c","Type":"ContainerStarted","Data":"a349f2924a8b1551454574be43e92453252fbef5a28173851b8781452b554f33"}
Oct 30 03:19:54.400239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.400200    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:19:54.400479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.400256    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p" event={"ID":"8c15638e-ea25-4fad-932c-5a614d44f05c","Type":"ContainerStarted","Data":"4958fcbb9cd0cca1642fd3160ecf29b0305b4cb4baba07ef1655525ea36abfd6"}
Oct 30 03:19:54.403785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.403744    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz" event={"ID":"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619","Type":"ContainerStarted","Data":"a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1"}
Oct 30 03:19:54.403913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.403823    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz" event={"ID":"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619","Type":"ContainerStarted","Data":"6a69b7d180e346a1180601d5b0a5ba2e5daceea52d0407f64d31a197da720ed5"}
Oct 30 03:19:54.404052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.404029    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:19:54.416365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.416304    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p" podStartSLOduration=2.416289548 podStartE2EDuration="2.416289548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:54.41164094 +0000 UTC m=+1022.840530358" watchObservedRunningTime="2026-10-30 03:19:54.416289548 +0000 UTC m=+1022.845178966"
Oct 30 03:19:54.450806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.450736    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz" podStartSLOduration=2.450715826 podStartE2EDuration="2.450715826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:54.449131367 +0000 UTC m=+1022.878020787" watchObservedRunningTime="2026-10-30 03:19:54.450715826 +0000 UTC m=+1022.879605245"
Oct 30 03:19:54.716731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.716629    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:54.951053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:54.951021    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:55.028706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:55.028675    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"]
Oct 30 03:19:55.032725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:55.032702    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ecb50c-4zft5"]
Oct 30 03:19:55.414555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:55.414477    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:19:55.703798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:19:55.703720    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f628a5d-93fe-491f-91cc-5d289264d75e" path="/var/lib/kubelet/pods/6f628a5d-93fe-491f-91cc-5d289264d75e/volumes"
Oct 30 03:20:03.864273 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.864179    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"]
Oct 30 03:20:03.869875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.869841    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:03.888503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.888478    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"]
Oct 30 03:20:03.896404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.896375    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cc0859-dockercfg-b8vqq\""
Oct 30 03:20:03.917722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.917686    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc0859\""
Oct 30 03:20:03.980741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.973231    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:03.980741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:03.973353    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5686\" (UniqueName: \"kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686\") pod \"payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:04.074051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:04.074020    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5686\" (UniqueName: \"kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686\") pod \"payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:04.074235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:04.074131    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:04.075094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:04.075067    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:04.092713 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:04.092683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5686\" (UniqueName: \"kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686\") pod \"payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:04.185094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:04.184998    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:04.495862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:04.495827    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"]
Oct 30 03:20:04.496750 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:04.496714    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b51ded_942d_417e_80b0_3c998ec09ef1.slice/crio-f5fc556f6e0b5740afa52cd800d3e42961ab161a3a0422cadff976b6d52a5e9d WatchSource:0}: Error finding container f5fc556f6e0b5740afa52cd800d3e42961ab161a3a0422cadff976b6d52a5e9d: Status 404 returned error can't find the container with id f5fc556f6e0b5740afa52cd800d3e42961ab161a3a0422cadff976b6d52a5e9d
Oct 30 03:20:05.416764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:05.416731    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:20:05.417277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:05.416819    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:20:05.506180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:05.506143    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5" event={"ID":"05b51ded-942d-417e-80b0-3c998ec09ef1","Type":"ContainerStarted","Data":"f12641dc9e3892e0303e675c8afc1896382a6d04eb9ba5bb3a107148c0374baa"}
Oct 30 03:20:05.506465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:05.506410    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5" event={"ID":"05b51ded-942d-417e-80b0-3c998ec09ef1","Type":"ContainerStarted","Data":"f5fc556f6e0b5740afa52cd800d3e42961ab161a3a0422cadff976b6d52a5e9d"}
Oct 30 03:20:05.506653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:05.506630    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:05.548562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:05.548510    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5" podStartSLOduration=2.54849667 podStartE2EDuration="2.54849667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:05.544971689 +0000 UTC m=+1033.973861118" watchObservedRunningTime="2026-10-30 03:20:05.54849667 +0000 UTC m=+1033.977386087"
Oct 30 03:20:06.028251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:06.028210    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"]
Oct 30 03:20:06.039621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:06.039590    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ba7fa48-sc4wt"]
Oct 30 03:20:07.700280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:07.700241    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f884eb0a-68a1-4fc5-b40d-af25919511b3" path="/var/lib/kubelet/pods/f884eb0a-68a1-4fc5-b40d-af25919511b3/volumes"
Oct 30 03:20:08.167629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.167587    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"]
Oct 30 03:20:08.179681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.177666    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.208829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.208778    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3b93b5-openshift-default-dockercfg-hfcx5\""
Oct 30 03:20:08.212724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.212693    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"]
Oct 30 03:20:08.224478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224440    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224488    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224623    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224678    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224721    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224809    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224856    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hn9p\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.224940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.224924    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.225031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.225011    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.326922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.326881    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.326949    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327040    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327074    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hn9p\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327143    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327225    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327525    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327693    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327686    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327750    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.327854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.327816    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.328134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.328114    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.328656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.328628    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.328777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.328757    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.329611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.329590    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.329874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.329854    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.347961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.347926    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hn9p\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.348199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.348174    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token\") pod \"e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") " pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.521153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.520073    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:08.876841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.876809    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"]
Oct 30 03:20:08.877982 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:08.877867    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b58b090_24eb_4272_bd9d_ddb332828f3e.slice/crio-59a336b3f39a0b382c5d8525d4ea3c313108ac952eb6ff6047dc3262553c2a87 WatchSource:0}: Error finding container 59a336b3f39a0b382c5d8525d4ea3c313108ac952eb6ff6047dc3262553c2a87: Status 404 returned error can't find the container with id 59a336b3f39a0b382c5d8525d4ea3c313108ac952eb6ff6047dc3262553c2a87
Oct 30 03:20:08.881873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.881838    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:08.881984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.881919    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:08.881984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:08.881959    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:09.544209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:09.544175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" event={"ID":"8b58b090-24eb-4272-bd9d-ddb332828f3e","Type":"ContainerStarted","Data":"5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"}
Oct 30 03:20:09.544376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:09.544229    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" event={"ID":"8b58b090-24eb-4272-bd9d-ddb332828f3e","Type":"ContainerStarted","Data":"59a336b3f39a0b382c5d8525d4ea3c313108ac952eb6ff6047dc3262553c2a87"}
Oct 30 03:20:09.599698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:09.599636    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podStartSLOduration=1.5996175510000001 podStartE2EDuration="1.599617551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:09.599560729 +0000 UTC m=+1038.028450159" watchObservedRunningTime="2026-10-30 03:20:09.599617551 +0000 UTC m=+1038.028506970"
Oct 30 03:20:10.521205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:10.521165    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:10.526966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:10.526936    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:10.551503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:10.551475    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:10.552396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:10.552377    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:20:14.050724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.050687    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"]
Oct 30 03:20:14.055233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.055217    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.082745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.082713    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3b93b5-dockercfg-vm4zn\""
Oct 30 03:20:14.082745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.082721    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3b93b5\""
Oct 30 03:20:14.088371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.088346    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"]
Oct 30 03:20:14.101357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.101315    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqf7p\" (UniqueName: \"kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p\") pod \"payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.101522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.101359    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.121334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.121306    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"]
Oct 30 03:20:14.126370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.126351    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.146832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.146806    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"]
Oct 30 03:20:14.203240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.203183    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbhv\" (UniqueName: \"kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv\") pod \"payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.218445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.216597    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqf7p\" (UniqueName: \"kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p\") pod \"payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.218445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.216647    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.218445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.216952    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.267407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.263513    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.291752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.290632    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqf7p\" (UniqueName: \"kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p\") pod \"payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") " pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.317839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.317754    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.317839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.317818    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbhv\" (UniqueName: \"kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv\") pod \"payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.318516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.318493    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.337801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.337774    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbhv\" (UniqueName: \"kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv\") pod \"payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.370067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.370035    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:14.446019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.441861    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:14.684712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.684669    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"]
Oct 30 03:20:14.684866 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:14.684784    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda717ddf0_ac5b_453c_8ac8_6ba1813d1752.slice/crio-729f562c48e9b10b34a6c5b7791fb32914fe60c1122ad614d06dddf0a290b9c2 WatchSource:0}: Error finding container 729f562c48e9b10b34a6c5b7791fb32914fe60c1122ad614d06dddf0a290b9c2: Status 404 returned error can't find the container with id 729f562c48e9b10b34a6c5b7791fb32914fe60c1122ad614d06dddf0a290b9c2
Oct 30 03:20:14.767780 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:14.767755    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35159481_21fc_4ce5_9c49_9050fe128365.slice/crio-7e9a40bdb4cd6fc6094500c3cd271403045e2c80c9eb116595c5ec35271c30bd WatchSource:0}: Error finding container 7e9a40bdb4cd6fc6094500c3cd271403045e2c80c9eb116595c5ec35271c30bd: Status 404 returned error can't find the container with id 7e9a40bdb4cd6fc6094500c3cd271403045e2c80c9eb116595c5ec35271c30bd
Oct 30 03:20:14.768977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:14.768955    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"]
Oct 30 03:20:15.631814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.631773    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz" event={"ID":"35159481-21fc-4ce5-9c49-9050fe128365","Type":"ContainerStarted","Data":"fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e"}
Oct 30 03:20:15.632300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.631862    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz" event={"ID":"35159481-21fc-4ce5-9c49-9050fe128365","Type":"ContainerStarted","Data":"7e9a40bdb4cd6fc6094500c3cd271403045e2c80c9eb116595c5ec35271c30bd"}
Oct 30 03:20:15.632550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.632529    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:15.634487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.634457    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9" event={"ID":"a717ddf0-ac5b-453c-8ac8-6ba1813d1752","Type":"ContainerStarted","Data":"d12a3cb87809a052947c83acd69d933df35be05d6d313f643f6e17f722f18d03"}
Oct 30 03:20:15.634624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.634504    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9" event={"ID":"a717ddf0-ac5b-453c-8ac8-6ba1813d1752","Type":"ContainerStarted","Data":"729f562c48e9b10b34a6c5b7791fb32914fe60c1122ad614d06dddf0a290b9c2"}
Oct 30 03:20:15.634624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.634591    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:15.673554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.673502    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz" podStartSLOduration=1.673488739 podStartE2EDuration="1.673488739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:15.670640355 +0000 UTC m=+1044.099529776" watchObservedRunningTime="2026-10-30 03:20:15.673488739 +0000 UTC m=+1044.102378157"
Oct 30 03:20:15.710110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:15.710052    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9" podStartSLOduration=1.710014723 podStartE2EDuration="1.710014723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:15.707877221 +0000 UTC m=+1044.136766640" watchObservedRunningTime="2026-10-30 03:20:15.710014723 +0000 UTC m=+1044.138904140"
Oct 30 03:20:16.235815 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:16.235772    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:20:16.517813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:16.517739    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:20:23.616226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:23.616188    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"]
Oct 30 03:20:26.644915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:26.644881    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:20:27.653068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:27.653035    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:20:28.616821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:28.616773    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerName="payload-processing" containerID="cri-o://417725000f9a430c42a4f693e9ed42d8d5de5e44af35b2437f64804960bcb815" gracePeriod=25
Oct 30 03:20:29.305463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.305413    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"]
Oct 30 03:20:29.310563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.310544    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.330191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.330157    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"]
Oct 30 03:20:29.338959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.336905    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:20:29.338959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.336929    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:20:29.434468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.431587    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.434468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.431668    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.434468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.431807    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr74t\" (UniqueName: \"kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.533160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.533117    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr74t\" (UniqueName: \"kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.533366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.533237    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.533366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.533270    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.533976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.533955    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.535737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.535715    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.553320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.553291    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr74t\" (UniqueName: \"kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t\") pod \"maas-api-revoke-sub-0cf9dc14-6vhjg\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.630242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.630159    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:29.954945 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:29.954911    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a0c923_ce8e_4f30_b88c_e828c0a2ba30.slice/crio-8958640f29a65499b63b51ff0bc1cf116ef29daa2ae21ef79fbd5d7e6aa63c3f WatchSource:0}: Error finding container 8958640f29a65499b63b51ff0bc1cf116ef29daa2ae21ef79fbd5d7e6aa63c3f: Status 404 returned error can't find the container with id 8958640f29a65499b63b51ff0bc1cf116ef29daa2ae21ef79fbd5d7e6aa63c3f
Oct 30 03:20:29.955290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:29.955264    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"]
Oct 30 03:20:30.514038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:30.513997    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:20:30.775567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:30.775534    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg" event={"ID":"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30","Type":"ContainerStarted","Data":"42e97b4775dc74621ef72f643b54988d7f6b2ea4bd096d37617a37179d20be1b"}
Oct 30 03:20:30.775723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:30.775593    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg" event={"ID":"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30","Type":"ContainerStarted","Data":"8958640f29a65499b63b51ff0bc1cf116ef29daa2ae21ef79fbd5d7e6aa63c3f"}
Oct 30 03:20:30.818229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:30.818179    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg" podStartSLOduration=1.8181665009999999 podStartE2EDuration="1.818166501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:30.81420852 +0000 UTC m=+1059.243097952" watchObservedRunningTime="2026-10-30 03:20:30.818166501 +0000 UTC m=+1059.247055919"
Oct 30 03:20:31.785627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:31.785592    4877 generic.go:356] "Generic (PLEG): container finished" podID="e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" containerID="42e97b4775dc74621ef72f643b54988d7f6b2ea4bd096d37617a37179d20be1b" exitCode=0
Oct 30 03:20:31.785997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:31.785663    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg" event={"ID":"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30","Type":"ContainerDied","Data":"42e97b4775dc74621ef72f643b54988d7f6b2ea4bd096d37617a37179d20be1b"}
Oct 30 03:20:32.963642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:32.963612    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:33.074830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.074750    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca\") pod \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") "
Oct 30 03:20:33.075003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.074900    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca\") pod \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") "
Oct 30 03:20:33.075003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.074977    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t\" (UniqueName: \"kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t\") pod \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\" (UID: \"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30\") "
Oct 30 03:20:33.075175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.075146    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca" pod "e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" (UID: "e5a0c923-ce8e-4f30-b88c-e828c0a2ba30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:33.075594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.075563    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:33.077241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.077215    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t" pod "e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" (UID: "e5a0c923-ce8e-4f30-b88c-e828c0a2ba30"). InnerVolumeSpecName "kube-api-access-nr74t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:33.080722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.080698    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca" pod "e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" (UID: "e5a0c923-ce8e-4f30-b88c-e828c0a2ba30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:33.177169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.177132    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:33.177169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.177165    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr74t\" (UniqueName: \"kubernetes.io/projected/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30-kube-api-access-nr74t\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:33.804208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.804171    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"
Oct 30 03:20:33.804208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.804200    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg" event={"ID":"e5a0c923-ce8e-4f30-b88c-e828c0a2ba30","Type":"ContainerDied","Data":"8958640f29a65499b63b51ff0bc1cf116ef29daa2ae21ef79fbd5d7e6aa63c3f"}
Oct 30 03:20:33.804502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:33.804250    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8958640f29a65499b63b51ff0bc1cf116ef29daa2ae21ef79fbd5d7e6aa63c3f"
Oct 30 03:20:40.160907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:40.160868    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"]
Oct 30 03:20:40.513736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:40.513638    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:20:41.475179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:41.475143    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"]
Oct 30 03:20:41.475596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:41.475357    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy" containerID="cri-o://4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f" gracePeriod=30
Oct 30 03:20:41.489170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:41.489132    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"]
Oct 30 03:20:41.489456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:41.489385    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy" containerID="cri-o://d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0" gracePeriod=30
Oct 30 03:20:43.918520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.918485    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"]
Oct 30 03:20:43.919694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.919635    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" containerName="revoke-keys"
Oct 30 03:20:43.919694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.919655    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" containerName="revoke-keys"
Oct 30 03:20:43.919905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.919888    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" containerName="revoke-keys"
Oct 30 03:20:43.923846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.923822    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:43.938626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.938599    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cc0859-openshift-default-dockercfg-8qzgs\""
Oct 30 03:20:43.956232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:43.956139    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"]
Oct 30 03:20:44.035251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035213    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035251    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665fk\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035333    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035450    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035499    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035516    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035558    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035676    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.035780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.035735    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137384    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665fk\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137472    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137525    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137565    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137587    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137639    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137795    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.137926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137893    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.138044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.137981    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.138044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.138019    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.138139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.138045    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.138139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.138079    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.138292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.138268    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.138349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.138272    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.140119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.140102    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.140332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.140315    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.156872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.156840    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.157026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.156999    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-665fk\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk\") pod \"e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") " pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.237529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.237419    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:44.573854 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:44.573822    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624f9351_9f13_49d9_9d99_aee4bf695eed.slice/crio-efd88d3518513251fb1858c93a1d414c551ee6648459c7d72ddddb5fc2cbfb84 WatchSource:0}: Error finding container efd88d3518513251fb1858c93a1d414c551ee6648459c7d72ddddb5fc2cbfb84: Status 404 returned error can't find the container with id efd88d3518513251fb1858c93a1d414c551ee6648459c7d72ddddb5fc2cbfb84
Oct 30 03:20:44.574248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.574225    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"]
Oct 30 03:20:44.576324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.576292    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:44.576412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.576354    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:44.576412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.576384    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:20:44.934997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.934957    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" event={"ID":"624f9351-9f13-49d9-9d99-aee4bf695eed","Type":"ContainerStarted","Data":"9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96"}
Oct 30 03:20:44.935468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:44.935035    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" event={"ID":"624f9351-9f13-49d9-9d99-aee4bf695eed","Type":"ContainerStarted","Data":"efd88d3518513251fb1858c93a1d414c551ee6648459c7d72ddddb5fc2cbfb84"}
Oct 30 03:20:45.161544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.161460    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq" podUID="0eff34af-416c-482c-a6ef-9fe68198d4a5" containerName="payload-processing" containerID="cri-o://60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874" gracePeriod=25
Oct 30 03:20:45.237909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.237869    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:45.239836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.239815    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:45.239903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.239864    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:45.528975 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.528941    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:20:45.572961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.572906    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podStartSLOduration=2.572891958 podStartE2EDuration="2.572891958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:45.085039156 +0000 UTC m=+1073.513928574" watchObservedRunningTime="2026-10-30 03:20:45.572891958 +0000 UTC m=+1074.001781381"
Oct 30 03:20:45.660485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.660395    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd\" (UniqueName: \"kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd\") pod \"0eff34af-416c-482c-a6ef-9fe68198d4a5\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") "
Oct 30 03:20:45.660677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.660627    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume\") pod \"0eff34af-416c-482c-a6ef-9fe68198d4a5\" (UID: \"0eff34af-416c-482c-a6ef-9fe68198d4a5\") "
Oct 30 03:20:45.661068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.661037    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume" pod "0eff34af-416c-482c-a6ef-9fe68198d4a5" (UID: "0eff34af-416c-482c-a6ef-9fe68198d4a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:45.661226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.661210    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eff34af-416c-482c-a6ef-9fe68198d4a5-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:45.662619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.662588    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd" pod "0eff34af-416c-482c-a6ef-9fe68198d4a5" (UID: "0eff34af-416c-482c-a6ef-9fe68198d4a5"). InnerVolumeSpecName "kube-api-access-4krwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:45.761981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.761943    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krwd\" (UniqueName: \"kubernetes.io/projected/0eff34af-416c-482c-a6ef-9fe68198d4a5-kube-api-access-4krwd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:45.944054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.944011    4877 generic.go:356] "Generic (PLEG): container finished" podID="0eff34af-416c-482c-a6ef-9fe68198d4a5" containerID="60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874" exitCode=0
Oct 30 03:20:45.944521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.944072    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"
Oct 30 03:20:45.944521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.944090    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq" event={"ID":"0eff34af-416c-482c-a6ef-9fe68198d4a5","Type":"ContainerDied","Data":"60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874"}
Oct 30 03:20:45.944521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.944175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq" event={"ID":"0eff34af-416c-482c-a6ef-9fe68198d4a5","Type":"ContainerDied","Data":"50217bfa832144568a27bc6f97d0fbdc870e74aa6b16642b30193d3fedd9ecf8"}
Oct 30 03:20:45.944521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.944200    4877 scope.go:122] "RemoveContainer" containerID="60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874"
Oct 30 03:20:45.954663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.954643    4877 scope.go:122] "RemoveContainer" containerID="60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874"
Oct 30 03:20:45.954918 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:45.954898    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874\": container with ID starting with 60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874 not found: ID does not exist" containerID="60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874"
Oct 30 03:20:45.954984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:45.954926    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874"} err="failed to get container status \"60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874\": rpc error: code = NotFound desc = could not find container \"60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874\": container with ID starting with 60ea20da646e97a248bede43dc2943b271b197d299e4cf87ba45109e355c4874 not found: ID does not exist"
Oct 30 03:20:46.014670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.014585    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"]
Oct 30 03:20:46.020846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.020810    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f247d1-79c5fb4d46-lwjlq"]
Oct 30 03:20:46.238106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.238075    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:46.238278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.238128    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:46.238396 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:46.238363    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:20:46.865718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.865687    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:20:46.922629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.922603    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:20:46.956388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.956354    4877 generic.go:356] "Generic (PLEG): container finished" podID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerID="d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0" exitCode=0
Oct 30 03:20:46.956896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.956585    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" event={"ID":"9865bd6e-64ed-4c02-9410-a58fd13cab8c","Type":"ContainerDied","Data":"d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0"}
Oct 30 03:20:46.956896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.956629    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv" event={"ID":"9865bd6e-64ed-4c02-9410-a58fd13cab8c","Type":"ContainerDied","Data":"90a1cc4c6bd5d48d4fd311d4af16589d9c216e7c6aee5c37c9d3f539817fd1cc"}
Oct 30 03:20:46.956896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.956651    4877 scope.go:122] "RemoveContainer" containerID="d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0"
Oct 30 03:20:46.956896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.956727    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"
Oct 30 03:20:46.959554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.959530    4877 generic.go:356] "Generic (PLEG): container finished" podID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerID="4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f" exitCode=0
Oct 30 03:20:46.960393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.960375    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"
Oct 30 03:20:46.960527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.960499    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" event={"ID":"713b5c0c-6af2-4b8b-a518-22df5d689146","Type":"ContainerDied","Data":"4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f"}
Oct 30 03:20:46.960592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.960546    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx" event={"ID":"713b5c0c-6af2-4b8b-a518-22df5d689146","Type":"ContainerDied","Data":"2c5605da665cd65b28612e3d47b4cebff399039eff506f4d676f43276297fffb"}
Oct 30 03:20:46.967598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.967581    4877 scope.go:122] "RemoveContainer" containerID="d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0"
Oct 30 03:20:46.967865 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:46.967847    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0\": container with ID starting with d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0 not found: ID does not exist" containerID="d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0"
Oct 30 03:20:46.967924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.967875    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0"} err="failed to get container status \"d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0\": rpc error: code = NotFound desc = could not find container \"d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0\": container with ID starting with d8a5ba562d3e13e493b2d403b16bbd60fc1d0a6dac80fbffa3d8748a31a9cbe0 not found: ID does not exist"
Oct 30 03:20:46.967924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.967899    4877 scope.go:122] "RemoveContainer" containerID="4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f"
Oct 30 03:20:46.991709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:46.991672    4877 scope.go:122] "RemoveContainer" containerID="fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"
Oct 30 03:20:47.004042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004008    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004056    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004094    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004120    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004171    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004195    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004232    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004293    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004329    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004367    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004414    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004470    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004526    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004588    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004655    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket\") pod \"713b5c0c-6af2-4b8b-a518-22df5d689146\" (UID: \"713b5c0c-6af2-4b8b-a518-22df5d689146\") "
Oct 30 03:20:47.004783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004696    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.004783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004724    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.005040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.004821    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket\") pod \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\" (UID: \"9865bd6e-64ed-4c02-9410-a58fd13cab8c\") "
Oct 30 03:20:47.022594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.022562    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.022797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.022774    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.023860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.022928    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.023860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.023031    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.023860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.023146    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.023860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.023481    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:47.023860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.023782    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.024359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.024334    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.025855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.025830    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:47.027932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.027748    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.027932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.027904    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.030131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.030106    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:47.032327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.032303    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "kube-api-access-b5s2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:47.032534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.032510    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token" pod "9865bd6e-64ed-4c02-9410-a58fd13cab8c" (UID: "9865bd6e-64ed-4c02-9410-a58fd13cab8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:47.032772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.032738    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:47.033078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.033057    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "kube-api-access-djrkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:47.033779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.033746    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:20:47.036690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.036665    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token" pod "713b5c0c-6af2-4b8b-a518-22df5d689146" (UID: "713b5c0c-6af2-4b8b-a518-22df5d689146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:47.040446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.040412    4877 scope.go:122] "RemoveContainer" containerID="4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f"
Oct 30 03:20:47.040790 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:47.040767    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f\": container with ID starting with 4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f not found: ID does not exist" containerID="4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f"
Oct 30 03:20:47.040866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.040796    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f"} err="failed to get container status \"4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f\": rpc error: code = NotFound desc = could not find container \"4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f\": container with ID starting with 4da9af028e163ce45e5e18df94c98b15b4e3582690d8cc96e25589cff771f24f not found: ID does not exist"
Oct 30 03:20:47.040866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.040816    4877 scope.go:122] "RemoveContainer" containerID="fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"
Oct 30 03:20:47.041151 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:47.041126    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877\": container with ID starting with fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877 not found: ID does not exist" containerID="fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"
Oct 30 03:20:47.041231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.041193    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877"} err="failed to get container status \"fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877\": rpc error: code = NotFound desc = could not find container \"fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877\": container with ID starting with fc9e51d257adaa1fbb9f2243761480e32e613fccc6b2525c57c9622d1806a877 not found: ID does not exist"
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106726    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5s2q\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-kube-api-access-b5s2q\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106773    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106792    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106807    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106822    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106836    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106853    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djrkt\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-kube-api-access-djrkt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.115522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106868    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.106882    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116120    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116138    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116157    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9865bd6e-64ed-4c02-9410-a58fd13cab8c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116174    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116191    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9865bd6e-64ed-4c02-9410-a58fd13cab8c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116206    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116221    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116235    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713b5c0c-6af2-4b8b-a518-22df5d689146-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.116298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.116252    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713b5c0c-6af2-4b8b-a518-22df5d689146-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:47.238814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.238776    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:47.239006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.238833    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:47.317202 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.317163    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"]
Oct 30 03:20:47.326633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.326572    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-99shv"]
Oct 30 03:20:47.361162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.361133    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"]
Oct 30 03:20:47.366970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.366899    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-zb6rx"]
Oct 30 03:20:47.716320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.716243    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eff34af-416c-482c-a6ef-9fe68198d4a5" path="/var/lib/kubelet/pods/0eff34af-416c-482c-a6ef-9fe68198d4a5/volumes"
Oct 30 03:20:47.716774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.716758    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" path="/var/lib/kubelet/pods/713b5c0c-6af2-4b8b-a518-22df5d689146/volumes"
Oct 30 03:20:47.717212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:47.717198    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" path="/var/lib/kubelet/pods/9865bd6e-64ed-4c02-9410-a58fd13cab8c/volumes"
Oct 30 03:20:48.238815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:48.238780    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:48.239241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:48.238855    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:49.073735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.071543    4877 generic.go:356] "Generic (PLEG): container finished" podID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerID="417725000f9a430c42a4f693e9ed42d8d5de5e44af35b2437f64804960bcb815" exitCode=0
Oct 30 03:20:49.073735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.071825    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" event={"ID":"5d90b18c-3b2e-4058-9d17-a0f33633c95b","Type":"ContainerDied","Data":"417725000f9a430c42a4f693e9ed42d8d5de5e44af35b2437f64804960bcb815"}
Oct 30 03:20:49.073735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.071886    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99" event={"ID":"5d90b18c-3b2e-4058-9d17-a0f33633c95b","Type":"ContainerDied","Data":"62fea93f30cbab51b62fab3544f3bdb7038247f4831adc892269e03d41a6c10b"}
Oct 30 03:20:49.073735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.071913    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62fea93f30cbab51b62fab3544f3bdb7038247f4831adc892269e03d41a6c10b"
Oct 30 03:20:49.073735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.073292    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:20:49.234194 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.234090    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume\") pod \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") "
Oct 30 03:20:49.238950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.238913    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume" pod "5d90b18c-3b2e-4058-9d17-a0f33633c95b" (UID: "5d90b18c-3b2e-4058-9d17-a0f33633c95b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:49.239845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.239820    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:49.240062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.240034    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:49.245950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.245926    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4\" (UniqueName: \"kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4\") pod \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\" (UID: \"5d90b18c-3b2e-4058-9d17-a0f33633c95b\") "
Oct 30 03:20:49.247398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.247373    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90b18c-3b2e-4058-9d17-a0f33633c95b-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:49.256793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.256763    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4" pod "5d90b18c-3b2e-4058-9d17-a0f33633c95b" (UID: "5d90b18c-3b2e-4058-9d17-a0f33633c95b"). InnerVolumeSpecName "kube-api-access-rzrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:49.348667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:49.348596    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzrj4\" (UniqueName: \"kubernetes.io/projected/5d90b18c-3b2e-4058-9d17-a0f33633c95b-kube-api-access-rzrj4\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:50.131450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.124943    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"
Oct 30 03:20:50.251833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.250772    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:50.251833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.250832    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:50.271511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.263243    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"]
Oct 30 03:20:50.297964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.297929    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-jsx99"]
Oct 30 03:20:50.951447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.951389    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"]
Oct 30 03:20:50.981450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:50.981307    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"]
Oct 30 03:20:51.304154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:51.300721    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 03:20:51.304154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:51.300814    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 03:20:51.830448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:51.820666    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" path="/var/lib/kubelet/pods/5d90b18c-3b2e-4058-9d17-a0f33633c95b/volumes"
Oct 30 03:20:52.096470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:52.096361    4877 scope.go:122] "RemoveContainer" containerID="9aebdb17474efa825948493070c96c85d2c0281a63707e4c54398ad500170529"
Oct 30 03:20:52.166616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:52.166496    4877 scope.go:122] "RemoveContainer" containerID="dd08d98296ff24771d4b8646b7072c1dcfbf731714bcd34547d32818ff4fdf5e"
Oct 30 03:20:52.387503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:52.377112    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:52.387503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:52.378609    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:52.447497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:52.416638    4877 scope.go:122] "RemoveContainer" containerID="811e5301b084e35aa2cd4ab59fa0d1b3ed4a428fd16f3977e2658909ae06f1b9"
Oct 30 03:20:52.460458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:52.460410    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:20:53.134509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:53.131093    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:20:53.498551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:53.481929    4877 scope.go:122] "RemoveContainer" containerID="4465fc01aa50bde03f51ba938c2eb6e176f2be8da323d81c5e8b9cc3e9e4b22b"
Oct 30 03:20:53.564690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:53.561146    4877 scope.go:122] "RemoveContainer" containerID="a886faff3dd2b992ef5e40b5a24a7d898c87a156c9397341ec2067acf180e16c"
Oct 30 03:20:55.431939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:55.431906    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"]
Oct 30 03:20:55.432572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:55.432536    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy" containerID="cri-o://3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181" gracePeriod=30
Oct 30 03:20:55.951953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:55.951899    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs" podUID="bdc03823-8f9c-4638-b09c-cea0777ceaea" containerName="payload-processing" containerID="cri-o://cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3" gracePeriod=25
Oct 30 03:20:55.992448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:55.990493    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx" podUID="ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" containerName="payload-pre-processing" containerID="cri-o://cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f" gracePeriod=25
Oct 30 03:20:56.327880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.327560    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:20:56.439112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.439081    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:20:56.447662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.447638    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume\") pod \"bdc03823-8f9c-4638-b09c-cea0777ceaea\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") "
Oct 30 03:20:56.447818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.447800    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7\" (UniqueName: \"kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7\") pod \"bdc03823-8f9c-4638-b09c-cea0777ceaea\" (UID: \"bdc03823-8f9c-4638-b09c-cea0777ceaea\") "
Oct 30 03:20:56.448181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.448155    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume" pod "bdc03823-8f9c-4638-b09c-cea0777ceaea" (UID: "bdc03823-8f9c-4638-b09c-cea0777ceaea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:56.448340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.448323    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc03823-8f9c-4638-b09c-cea0777ceaea-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.449817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.449793    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7" pod "bdc03823-8f9c-4638-b09c-cea0777ceaea" (UID: "bdc03823-8f9c-4638-b09c-cea0777ceaea"). InnerVolumeSpecName "kube-api-access-tw7w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:56.549820 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.549788    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds\" (UniqueName: \"kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds\") pod \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") "
Oct 30 03:20:56.549963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.549871    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume\") pod \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\" (UID: \"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c\") "
Oct 30 03:20:56.550233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.550220    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw7w7\" (UniqueName: \"kubernetes.io/projected/bdc03823-8f9c-4638-b09c-cea0777ceaea-kube-api-access-tw7w7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.550546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.550528    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume" pod "ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" (UID: "ba529ec9-9a27-4ff7-8539-c09cfcea4c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:20:56.552714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.552683    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds" pod "ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" (UID: "ba529ec9-9a27-4ff7-8539-c09cfcea4c5c"). InnerVolumeSpecName "kube-api-access-ppdds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:56.651774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.651746    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppdds\" (UniqueName: \"kubernetes.io/projected/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-kube-api-access-ppdds\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.651774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.651771    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:20:56.929387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.929280    4877 generic.go:356] "Generic (PLEG): container finished" podID="bdc03823-8f9c-4638-b09c-cea0777ceaea" containerID="cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3" exitCode=0
Oct 30 03:20:56.929387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.929367    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"
Oct 30 03:20:56.929618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.929382    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs" event={"ID":"bdc03823-8f9c-4638-b09c-cea0777ceaea","Type":"ContainerDied","Data":"cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3"}
Oct 30 03:20:56.929618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.929488    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs" event={"ID":"bdc03823-8f9c-4638-b09c-cea0777ceaea","Type":"ContainerDied","Data":"f8e4d58559264fc084ba8784b5a80dadd6d43c8b8332f690ea400201feef1618"}
Oct 30 03:20:56.929618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.929561    4877 scope.go:122] "RemoveContainer" containerID="cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3"
Oct 30 03:20:56.931119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.930938    4877 generic.go:356] "Generic (PLEG): container finished" podID="ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" containerID="cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f" exitCode=0
Oct 30 03:20:56.931119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.930991    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"
Oct 30 03:20:56.931119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.930989    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx" event={"ID":"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c","Type":"ContainerDied","Data":"cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f"}
Oct 30 03:20:56.931119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.931094    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx" event={"ID":"ba529ec9-9a27-4ff7-8539-c09cfcea4c5c","Type":"ContainerDied","Data":"12ec2019e73a26378923f46aa5e5d7f45cfe137217495a352973753270636493"}
Oct 30 03:20:56.959653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.959626    4877 scope.go:122] "RemoveContainer" containerID="cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3"
Oct 30 03:20:56.961323 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:56.961298    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3\": container with ID starting with cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3 not found: ID does not exist" containerID="cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3"
Oct 30 03:20:56.961412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.961329    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3"} err="failed to get container status \"cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3\": rpc error: code = NotFound desc = could not find container \"cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3\": container with ID starting with cdba572eb346d24c034ff22aee850f581577f4c83ace6393368d2d251be114c3 not found: ID does not exist"
Oct 30 03:20:56.961412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.961346    4877 scope.go:122] "RemoveContainer" containerID="cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f"
Oct 30 03:20:56.989655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.989630    4877 scope.go:122] "RemoveContainer" containerID="cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f"
Oct 30 03:20:56.990117 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:20:56.990089    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f\": container with ID starting with cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f not found: ID does not exist" containerID="cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f"
Oct 30 03:20:56.990220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.990126    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f"} err="failed to get container status \"cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f\": rpc error: code = NotFound desc = could not find container \"cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f\": container with ID starting with cf65f699e5576f4d902a2d55d821dc1ce3d0c700809600bbbd5d335cc6745d3f not found: ID does not exist"
Oct 30 03:20:56.997822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:56.997788    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"]
Oct 30 03:20:57.010091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:57.010059    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-585938-5495967d7f-wkxdx"]
Oct 30 03:20:57.039411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:57.039376    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"]
Oct 30 03:20:57.043936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:57.043906    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-585938-d67fbcff4-d7fhs"]
Oct 30 03:20:57.699245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:57.699217    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" path="/var/lib/kubelet/pods/ba529ec9-9a27-4ff7-8539-c09cfcea4c5c/volumes"
Oct 30 03:20:57.699627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:57.699613    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc03823-8f9c-4638-b09c-cea0777ceaea" path="/var/lib/kubelet/pods/bdc03823-8f9c-4638-b09c-cea0777ceaea/volumes"
Oct 30 03:20:58.889247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:58.889208    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"]
Oct 30 03:20:58.889844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:58.889472    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy" containerID="cri-o://e16320c475f095385e86a29b52800aa6aeae8d9af2154e1a90bb66f9ab1c2e5e" gracePeriod=30
Oct 30 03:20:59.035364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035321    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"]
Oct 30 03:20:59.035935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035915    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerName="payload-processing"
Oct 30 03:20:59.035935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035933    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerName="payload-processing"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035954    4877 cpu_manager.go:414] "containerMap: removing container" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035960    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035981    4877 cpu_manager.go:414] "containerMap: removing container" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.035986    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036000    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0eff34af-416c-482c-a6ef-9fe68198d4a5" containerName="payload-processing"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036006    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eff34af-416c-482c-a6ef-9fe68198d4a5" containerName="payload-processing"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036019    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" containerName="payload-pre-processing"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036024    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" containerName="payload-pre-processing"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036032    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036037    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036043    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bdc03823-8f9c-4638-b09c-cea0777ceaea" containerName="payload-processing"
Oct 30 03:20:59.036106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036048    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc03823-8f9c-4638-b09c-cea0777ceaea" containerName="payload-processing"
Oct 30 03:20:59.036719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036157    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d90b18c-3b2e-4058-9d17-a0f33633c95b" containerName="payload-processing"
Oct 30 03:20:59.036719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036168    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy"
Oct 30 03:20:59.036719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036178    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eff34af-416c-482c-a6ef-9fe68198d4a5" containerName="payload-processing"
Oct 30 03:20:59.036719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036186    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba529ec9-9a27-4ff7-8539-c09cfcea4c5c" containerName="payload-pre-processing"
Oct 30 03:20:59.036719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036193    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9865bd6e-64ed-4c02-9410-a58fd13cab8c" containerName="istio-proxy"
Oct 30 03:20:59.036719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.036199    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc03823-8f9c-4638-b09c-cea0777ceaea" containerName="payload-processing"
Oct 30 03:20:59.041344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.041323    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.078362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.078335    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b132360f-dockercfg-gcrp5\""
Oct 30 03:20:59.079901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.079470    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b132360f\""
Oct 30 03:20:59.084544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.084510    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"]
Oct 30 03:20:59.086478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.086417    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnvr\" (UniqueName: \"kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.086648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.086560    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.115467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.111914    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"]
Oct 30 03:20:59.115467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.112994    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="713b5c0c-6af2-4b8b-a518-22df5d689146" containerName="istio-proxy"
Oct 30 03:20:59.117066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.117032    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.188964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.187668    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrhv\" (UniqueName: \"kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.188964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.187734    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnvr\" (UniqueName: \"kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.188964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.187778    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.188964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.187835    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.189294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.189159    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.206444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.198507    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"]
Oct 30 03:20:59.241861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.241751    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnvr\" (UniqueName: \"kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.290134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.290083    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.290328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.290310    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrhv\" (UniqueName: \"kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.291856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.291810    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.315142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.315105    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrhv\" (UniqueName: \"kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv\") pod \"payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") " pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.355827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.355796    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.461496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.456450    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:20:59.672837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.672810    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"]
Oct 30 03:20:59.674119 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:59.674083    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5a4c67_d88e_4bda_9fa2_ea6393f074b8.slice/crio-f7c8a6a96fbbe49851fe91d1d809817836358a8f2cb5b470247cddf3ec1ce870 WatchSource:0}: Error finding container f7c8a6a96fbbe49851fe91d1d809817836358a8f2cb5b470247cddf3ec1ce870: Status 404 returned error can't find the container with id f7c8a6a96fbbe49851fe91d1d809817836358a8f2cb5b470247cddf3ec1ce870
Oct 30 03:20:59.863215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.862557    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"]
Oct 30 03:20:59.865679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:20:59.865652    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb73ac0a_d179_41ed_a82e_f182b7167e60.slice/crio-0e7aca5bdf9161e4dcd91060fa88b2f9fcf9892da9883789a36076c32a31c9a4 WatchSource:0}: Error finding container 0e7aca5bdf9161e4dcd91060fa88b2f9fcf9892da9883789a36076c32a31c9a4: Status 404 returned error can't find the container with id 0e7aca5bdf9161e4dcd91060fa88b2f9fcf9892da9883789a36076c32a31c9a4
Oct 30 03:20:59.966535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.965926    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" event={"ID":"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8","Type":"ContainerStarted","Data":"7cd00a0c8c42c1a73257d0c6890d63df6336227d2af6866eb6ebcca58a121ff0"}
Oct 30 03:20:59.966535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.966028    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" event={"ID":"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8","Type":"ContainerStarted","Data":"f7c8a6a96fbbe49851fe91d1d809817836358a8f2cb5b470247cddf3ec1ce870"}
Oct 30 03:20:59.966535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.966061    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:20:59.967867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.967847    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" event={"ID":"bb73ac0a-d179-41ed-a82e-f182b7167e60","Type":"ContainerStarted","Data":"deb7292adfaeee5c62ac4b4884039760ce5f622e4dc5bed2a18c379f6e0e32db"}
Oct 30 03:20:59.967947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.967884    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" event={"ID":"bb73ac0a-d179-41ed-a82e-f182b7167e60","Type":"ContainerStarted","Data":"0e7aca5bdf9161e4dcd91060fa88b2f9fcf9892da9883789a36076c32a31c9a4"}
Oct 30 03:20:59.967996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:20:59.967983    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:21:00.008731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.008673    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" podStartSLOduration=2.00865527 podStartE2EDuration="2.00865527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:00.004271913 +0000 UTC m=+1088.433161331" watchObservedRunningTime="2026-10-30 03:21:00.00865527 +0000 UTC m=+1088.437544690"
Oct 30 03:21:00.051259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.051190    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" podStartSLOduration=1.051170202 podStartE2EDuration="1.051170202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:00.047280487 +0000 UTC m=+1088.476169922" watchObservedRunningTime="2026-10-30 03:21:00.051170202 +0000 UTC m=+1088.480059617"
Oct 30 03:21:00.709921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.709858    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:21:00.806768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.806722    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.806957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.806800    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.806957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.806862    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.806957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.806912    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.807120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.806990    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.807120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.807018    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.807120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.807081    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.807271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.807153    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.807271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.807206    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data\") pod \"ca5ad99e-329e-4a1e-acfb-da8960421f17\" (UID: \"ca5ad99e-329e-4a1e-acfb-da8960421f17\") "
Oct 30 03:21:00.808400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.808370    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:00.808800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.808775    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:00.809176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.809134    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:00.809852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.809795    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:00.809955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.809864    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:00.810850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.810823    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:00.812155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.812125    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "kube-api-access-llkmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:00.812299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.812274    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:00.812393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.812375    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo" pod "ca5ad99e-329e-4a1e-acfb-da8960421f17" (UID: "ca5ad99e-329e-4a1e-acfb-da8960421f17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:21:00.908493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908457    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkmg\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-kube-api-access-llkmg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908488    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908499    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908508    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908518    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908527    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908536    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908544    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca5ad99e-329e-4a1e-acfb-da8960421f17-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.908715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.908552    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca5ad99e-329e-4a1e-acfb-da8960421f17-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:00.978115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.978052    4877 generic.go:356] "Generic (PLEG): container finished" podID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerID="3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181" exitCode=0
Oct 30 03:21:00.981385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.978124    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"
Oct 30 03:21:00.981385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.978140    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" event={"ID":"ca5ad99e-329e-4a1e-acfb-da8960421f17","Type":"ContainerDied","Data":"3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181"}
Oct 30 03:21:00.981385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.978199    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk" event={"ID":"ca5ad99e-329e-4a1e-acfb-da8960421f17","Type":"ContainerDied","Data":"300513df7277ea294b6bfbc99b1504b891e67d03f05edcf982b0f279832c8616"}
Oct 30 03:21:00.981385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.978221    4877 scope.go:122] "RemoveContainer" containerID="3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181"
Oct 30 03:21:00.989742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:00.989724    4877 scope.go:122] "RemoveContainer" containerID="a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"
Oct 30 03:21:01.006312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.006291    4877 scope.go:122] "RemoveContainer" containerID="3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181"
Oct 30 03:21:01.006598 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:01.006576    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181\": container with ID starting with 3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181 not found: ID does not exist" containerID="3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181"
Oct 30 03:21:01.006692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.006604    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181"} err="failed to get container status \"3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181\": rpc error: code = NotFound desc = could not find container \"3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181\": container with ID starting with 3c0b5d1c323c0247195d3ec0de1f517b0e819a051b82d14590f814a976b67181 not found: ID does not exist"
Oct 30 03:21:01.006692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.006632    4877 scope.go:122] "RemoveContainer" containerID="a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"
Oct 30 03:21:01.006966 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:01.006945    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b\": container with ID starting with a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b not found: ID does not exist" containerID="a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"
Oct 30 03:21:01.007012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.006970    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b"} err="failed to get container status \"a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b\": rpc error: code = NotFound desc = could not find container \"a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b\": container with ID starting with a0f4eefbd3aa2fef827b734f51a8beeecd3cf14c8a59c0e51160920769ac4b2b not found: ID does not exist"
Oct 30 03:21:01.031126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.031099    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"]
Oct 30 03:21:01.034353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.034331    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f247d1-openshift-default-6b77d647bf-p24vk"]
Oct 30 03:21:01.503777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.503685    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"]
Oct 30 03:21:01.541314 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.541277    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"]
Oct 30 03:21:01.698784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.698751    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" path="/var/lib/kubelet/pods/ca5ad99e-329e-4a1e-acfb-da8960421f17/volumes"
Oct 30 03:21:01.986817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.986786    4877 generic.go:356] "Generic (PLEG): container finished" podID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerID="5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf" exitCode=0
Oct 30 03:21:01.987265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.986860    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" event={"ID":"8b58b090-24eb-4272-bd9d-ddb332828f3e","Type":"ContainerDied","Data":"5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"}
Oct 30 03:21:01.987331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.987287    4877 scope.go:122] "RemoveContainer" containerID="5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"
Oct 30 03:21:01.988818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.988788    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:01.988948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.988881    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:01.988948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:01.988929    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:03.009948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:03.009911    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" event={"ID":"8b58b090-24eb-4272-bd9d-ddb332828f3e","Type":"ContainerStarted","Data":"448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c"}
Oct 30 03:21:03.520513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:03.520471    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:21:03.520771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:03.520741    4877 patch_prober.go:29] interesting pod/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:21:03.520822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:03.520804    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 03:21:04.022929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.022890    4877 generic.go:356] "Generic (PLEG): container finished" podID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerID="e16320c475f095385e86a29b52800aa6aeae8d9af2154e1a90bb66f9ab1c2e5e" exitCode=0
Oct 30 03:21:04.023316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.022971    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" event={"ID":"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe","Type":"ContainerDied","Data":"e16320c475f095385e86a29b52800aa6aeae8d9af2154e1a90bb66f9ab1c2e5e"}
Oct 30 03:21:04.522334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.521900    4877 patch_prober.go:29] interesting pod/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:21:04.522334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.521959    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 03:21:04.625166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.624819    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:21:04.663672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663593    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.663833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663675    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.663833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663738    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.663833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663803    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.664008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663834    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.664008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663928    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.664008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663977    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:04.664171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.663991    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.664171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664087    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.664171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664075    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:04.664171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664132    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket\") pod \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\" (UID: \"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe\") "
Oct 30 03:21:04.664171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664143    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:04.664453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664406    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:04.664519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664457    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:04.664743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664719    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.664878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664747    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.664878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664764    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.664878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664778    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.664878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.664791    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.666280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.666259    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:21:04.666405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.666389    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:04.666694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.666542    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "kube-api-access-9hmw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:04.666694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.666598    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token" pod "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" (UID: "b4b5a35b-1de8-4afe-8dd5-309dd8a95efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:04.766079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.766044    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.766079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.766072    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.766079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.766081    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hmw2\" (UniqueName: \"kubernetes.io/projected/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-kube-api-access-9hmw2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.766304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.766090    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:04.968013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.967929    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"]
Oct 30 03:21:04.968596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968580    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy"
Oct 30 03:21:04.968661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968598    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy"
Oct 30 03:21:04.968661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968650    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy"
Oct 30 03:21:04.968727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968659    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy"
Oct 30 03:21:04.968763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968753    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy"
Oct 30 03:21:04.968801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968768    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" containerName="istio-proxy"
Oct 30 03:21:04.968801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.968776    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy"
Oct 30 03:21:04.972583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.972566    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:04.996084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:04.995200    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d4b0d5\""
Oct 30 03:21:05.023006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.022922    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"]
Oct 30 03:21:05.038844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.038818    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p" event={"ID":"b4b5a35b-1de8-4afe-8dd5-309dd8a95efe","Type":"ContainerDied","Data":"7c20052202ab9e564145fd4c70f6906550ac806113635f20c398b4efb93badd4"}
Oct 30 03:21:05.038981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.038871    4877 scope.go:122] "RemoveContainer" containerID="e16320c475f095385e86a29b52800aa6aeae8d9af2154e1a90bb66f9ab1c2e5e"
Oct 30 03:21:05.038981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.038974    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"
Oct 30 03:21:05.042360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.042334    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"]
Oct 30 03:21:05.043401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.043376    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy"
Oct 30 03:21:05.043530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.043403    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5ad99e-329e-4a1e-acfb-da8960421f17" containerName="istio-proxy"
Oct 30 03:21:05.047956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.047931    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.063042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.063017    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"]
Oct 30 03:21:05.063467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.063413    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d4b0d5-dockercfg-mfr2b\""
Oct 30 03:21:05.069168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.069140    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.069625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.069463    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4c2\" (UniqueName: \"kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2\") pod \"payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.069625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.069498    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4tnd\" (UniqueName: \"kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd\") pod \"payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.069625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.069544    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.172135    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4c2\" (UniqueName: \"kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2\") pod \"payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.172197    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4tnd\" (UniqueName: \"kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd\") pod \"payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.172271    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.172352    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.173290    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.175442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.174512    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.220134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.219993    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4c2\" (UniqueName: \"kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2\") pod \"payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.233723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.233624    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4tnd\" (UniqueName: \"kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd\") pod \"payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.236840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.236777    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"]
Oct 30 03:21:05.243661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.243636    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-j7m5p"]
Oct 30 03:21:05.335369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.335333    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:05.369023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.368982    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:05.520714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.520682    4877 patch_prober.go:29] interesting pod/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:21:05.520863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.520744    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 03:21:05.728653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.725846    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b5a35b-1de8-4afe-8dd5-309dd8a95efe" path="/var/lib/kubelet/pods/b4b5a35b-1de8-4afe-8dd5-309dd8a95efe/volumes"
Oct 30 03:21:05.728653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.726302    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"]
Oct 30 03:21:05.766227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.766183    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"]
Oct 30 03:21:05.767737 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:21:05.767708    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde1591a_2b1e_4c45_b5a5_06f0145e00f6.slice/crio-f58cc12ca61e41751f038ce3960a56c922c41d5f28598ac58b2366b10cfc1000 WatchSource:0}: Error finding container f58cc12ca61e41751f038ce3960a56c922c41d5f28598ac58b2366b10cfc1000: Status 404 returned error can't find the container with id f58cc12ca61e41751f038ce3960a56c922c41d5f28598ac58b2366b10cfc1000
Oct 30 03:21:05.997651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:05.997610    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"]
Oct 30 03:21:06.050898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.050863    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf" event={"ID":"fde1591a-2b1e-4c45-b5a5-06f0145e00f6","Type":"ContainerStarted","Data":"6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180"}
Oct 30 03:21:06.051390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.050927    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf" event={"ID":"fde1591a-2b1e-4c45-b5a5-06f0145e00f6","Type":"ContainerStarted","Data":"f58cc12ca61e41751f038ce3960a56c922c41d5f28598ac58b2366b10cfc1000"}
Oct 30 03:21:06.051390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.051068    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:06.052646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.052622    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c" event={"ID":"1a6038ad-db04-4cf3-88d1-874a87a7776d","Type":"ContainerStarted","Data":"1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82"}
Oct 30 03:21:06.052772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.052666    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:06.052772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.052691    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c" event={"ID":"1a6038ad-db04-4cf3-88d1-874a87a7776d","Type":"ContainerStarted","Data":"7eac89d8d3572b8396286d96f8bf7ca426de2e25beefd824662fed8440e3978f"}
Oct 30 03:21:06.101452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.094928    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf" podStartSLOduration=1.094908562 podStartE2EDuration="1.094908562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:06.092794952 +0000 UTC m=+1094.521684389" watchObservedRunningTime="2026-10-30 03:21:06.094908562 +0000 UTC m=+1094.523797981"
Oct 30 03:21:06.507998 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.507900    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz" podUID="cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" containerName="payload-processing" containerID="cri-o://a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1" gracePeriod=25
Oct 30 03:21:06.521135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.521098    4877 patch_prober.go:29] interesting pod/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:21:06.521251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.521154    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 03:21:06.541718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.541686    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p" podUID="8c15638e-ea25-4fad-932c-5a614d44f05c" containerName="payload-pre-processing" containerID="cri-o://a349f2924a8b1551454574be43e92453252fbef5a28173851b8781452b554f33" gracePeriod=25
Oct 30 03:21:06.974779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:06.974738    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:21:07.037787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.037723    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c" podStartSLOduration=3.037704656 podStartE2EDuration="3.037704656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:06.146550223 +0000 UTC m=+1094.575439643" watchObservedRunningTime="2026-10-30 03:21:07.037704656 +0000 UTC m=+1095.466594074"
Oct 30 03:21:07.051448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.049826    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4\" (UniqueName: \"kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4\") pod \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") "
Oct 30 03:21:07.051448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.049937    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume\") pod \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\" (UID: \"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619\") "
Oct 30 03:21:07.051448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.050803    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume" pod "cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" (UID: "cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:07.072582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.060593    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4" pod "cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" (UID: "cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619"). InnerVolumeSpecName "kube-api-access-prkh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.142229    4877 generic.go:356] "Generic (PLEG): container finished" podID="8c15638e-ea25-4fad-932c-5a614d44f05c" containerID="a349f2924a8b1551454574be43e92453252fbef5a28173851b8781452b554f33" exitCode=0
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.142352    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p" event={"ID":"8c15638e-ea25-4fad-932c-5a614d44f05c","Type":"ContainerDied","Data":"a349f2924a8b1551454574be43e92453252fbef5a28173851b8781452b554f33"}
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.143980    4877 generic.go:356] "Generic (PLEG): container finished" podID="cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" containerID="a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1" exitCode=0
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.145034    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.148789    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz" event={"ID":"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619","Type":"ContainerDied","Data":"a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1"}
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.148850    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz" event={"ID":"cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619","Type":"ContainerDied","Data":"6a69b7d180e346a1180601d5b0a5ba2e5daceea52d0407f64d31a197da720ed5"}
Oct 30 03:21:07.149494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.148890    4877 scope.go:122] "RemoveContainer" containerID="a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1"
Oct 30 03:21:07.153442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.153008    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkh4\" (UniqueName: \"kubernetes.io/projected/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-kube-api-access-prkh4\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:07.153442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.153058    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:07.169616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.169579    4877 scope.go:122] "RemoveContainer" containerID="a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1"
Oct 30 03:21:07.169924 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:07.169851    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1\": container with ID starting with a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1 not found: ID does not exist" containerID="a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1"
Oct 30 03:21:07.169924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.169878    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1"} err="failed to get container status \"a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1\": rpc error: code = NotFound desc = could not find container \"a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1\": container with ID starting with a59c304e66f229b298d1c4c7228cf67f0baa017485e29a6ae617886d4fb762e1 not found: ID does not exist"
Oct 30 03:21:07.172862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.172837    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:21:07.220182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.220152    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"]
Oct 30 03:21:07.224879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.224842    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e4daf-5ccf85944b-d26rz"]
Oct 30 03:21:07.254453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.254400    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd\" (UniqueName: \"kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd\") pod \"8c15638e-ea25-4fad-932c-5a614d44f05c\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") "
Oct 30 03:21:07.254778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.254758    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume\") pod \"8c15638e-ea25-4fad-932c-5a614d44f05c\" (UID: \"8c15638e-ea25-4fad-932c-5a614d44f05c\") "
Oct 30 03:21:07.255919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.255893    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume" pod "8c15638e-ea25-4fad-932c-5a614d44f05c" (UID: "8c15638e-ea25-4fad-932c-5a614d44f05c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:07.258030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.257998    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd" pod "8c15638e-ea25-4fad-932c-5a614d44f05c" (UID: "8c15638e-ea25-4fad-932c-5a614d44f05c"). InnerVolumeSpecName "kube-api-access-47vfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:07.368528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.363975    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vfd\" (UniqueName: \"kubernetes.io/projected/8c15638e-ea25-4fad-932c-5a614d44f05c-kube-api-access-47vfd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:07.368528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.364008    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15638e-ea25-4fad-932c-5a614d44f05c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:07.520644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.520608    4877 patch_prober.go:29] interesting pod/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 03:21:07.520839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.520674    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 03:21:07.718614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:07.718531    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" path="/var/lib/kubelet/pods/cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619/volumes"
Oct 30 03:21:08.153208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.153176    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"
Oct 30 03:21:08.153766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.153217    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p" event={"ID":"8c15638e-ea25-4fad-932c-5a614d44f05c","Type":"ContainerDied","Data":"4958fcbb9cd0cca1642fd3160ecf29b0305b4cb4baba07ef1655525ea36abfd6"}
Oct 30 03:21:08.153766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.153286    4877 scope.go:122] "RemoveContainer" containerID="a349f2924a8b1551454574be43e92453252fbef5a28173851b8781452b554f33"
Oct 30 03:21:08.207647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.207611    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"]
Oct 30 03:21:08.218946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.218916    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e4daf-6cbfb4d5d7-xz92p"]
Oct 30 03:21:08.525098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.525066    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:21:08.525556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.525451    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:21:08.526271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:08.526252    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:21:09.704079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:09.704039    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c15638e-ea25-4fad-932c-5a614d44f05c" path="/var/lib/kubelet/pods/8c15638e-ea25-4fad-932c-5a614d44f05c/volumes"
Oct 30 03:21:10.981832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:10.981804    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:21:10.982399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:10.981888    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:21:10.998647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:10.998612    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5" podUID="05b51ded-942d-417e-80b0-3c998ec09ef1" containerName="payload-pre-processing" containerID="cri-o://f12641dc9e3892e0303e675c8afc1896382a6d04eb9ba5bb3a107148c0374baa" gracePeriod=25
Oct 30 03:21:11.189548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.185682    4877 generic.go:356] "Generic (PLEG): container finished" podID="05b51ded-942d-417e-80b0-3c998ec09ef1" containerID="f12641dc9e3892e0303e675c8afc1896382a6d04eb9ba5bb3a107148c0374baa" exitCode=0
Oct 30 03:21:11.189548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.185777    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5" event={"ID":"05b51ded-942d-417e-80b0-3c998ec09ef1","Type":"ContainerDied","Data":"f12641dc9e3892e0303e675c8afc1896382a6d04eb9ba5bb3a107148c0374baa"}
Oct 30 03:21:11.313649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.313620    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:21:11.422865 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.422834    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686\" (UniqueName: \"kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686\") pod \"05b51ded-942d-417e-80b0-3c998ec09ef1\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") "
Oct 30 03:21:11.423114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.423096    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume\") pod \"05b51ded-942d-417e-80b0-3c998ec09ef1\" (UID: \"05b51ded-942d-417e-80b0-3c998ec09ef1\") "
Oct 30 03:21:11.424593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.424564    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume" pod "05b51ded-942d-417e-80b0-3c998ec09ef1" (UID: "05b51ded-942d-417e-80b0-3c998ec09ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:11.432477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.431060    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686" pod "05b51ded-942d-417e-80b0-3c998ec09ef1" (UID: "05b51ded-942d-417e-80b0-3c998ec09ef1"). InnerVolumeSpecName "kube-api-access-v5686". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:11.525378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.525345    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5686\" (UniqueName: \"kubernetes.io/projected/05b51ded-942d-417e-80b0-3c998ec09ef1-kube-api-access-v5686\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:11.525378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:11.525373    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05b51ded-942d-417e-80b0-3c998ec09ef1-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:12.196333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:12.196272    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5" event={"ID":"05b51ded-942d-417e-80b0-3c998ec09ef1","Type":"ContainerDied","Data":"f5fc556f6e0b5740afa52cd800d3e42961ab161a3a0422cadff976b6d52a5e9d"}
Oct 30 03:21:12.196825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:12.196376    4877 scope.go:122] "RemoveContainer" containerID="f12641dc9e3892e0303e675c8afc1896382a6d04eb9ba5bb3a107148c0374baa"
Oct 30 03:21:12.196825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:12.196299    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"
Oct 30 03:21:12.249412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:12.249381    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"]
Oct 30 03:21:12.260939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:12.260910    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cc0859-58898cb84c-dk9q5"]
Oct 30 03:21:13.701443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:13.701345    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b51ded-942d-417e-80b0-3c998ec09ef1" path="/var/lib/kubelet/pods/05b51ded-942d-417e-80b0-3c998ec09ef1/volumes"
Oct 30 03:21:16.232459 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:16.232385    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:21:17.149350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:17.149129    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:21:17.150778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:17.150758    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:21:19.058604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.058571    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"]
Oct 30 03:21:19.059211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059193    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" containerName="payload-processing"
Oct 30 03:21:19.059254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059214    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" containerName="payload-processing"
Oct 30 03:21:19.059254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059238    4877 cpu_manager.go:414] "containerMap: removing container" podUID="05b51ded-942d-417e-80b0-3c998ec09ef1" containerName="payload-pre-processing"
Oct 30 03:21:19.059254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059243    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b51ded-942d-417e-80b0-3c998ec09ef1" containerName="payload-pre-processing"
Oct 30 03:21:19.059351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059263    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8c15638e-ea25-4fad-932c-5a614d44f05c" containerName="payload-pre-processing"
Oct 30 03:21:19.059351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059268    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c15638e-ea25-4fad-932c-5a614d44f05c" containerName="payload-pre-processing"
Oct 30 03:21:19.059415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059364    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c15638e-ea25-4fad-932c-5a614d44f05c" containerName="payload-pre-processing"
Oct 30 03:21:19.059415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059377    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd7f9e2-7f68-4d1d-8f33-ca65ae8dc619" containerName="payload-processing"
Oct 30 03:21:19.059415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.059386    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b51ded-942d-417e-80b0-3c998ec09ef1" containerName="payload-pre-processing"
Oct 30 03:21:19.064368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.064341    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.088378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.088338    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"]
Oct 30 03:21:19.127649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.127604    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.127790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.127654    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4hd\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.127848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.127804    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.127889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.127857    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.127972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.127947    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.128053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.128034    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.128106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.128068    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.128161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.128141    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.128305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.128286    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229121    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229228    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229249    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4hd\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229298    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229332    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229365    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229457    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229495    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229576    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229816    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229852    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.229946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.229896    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.230110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.230085    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.230256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.230239    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.231886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.231865    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.232208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.232186    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.250199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.250169    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.250344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.250328    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4hd\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6vq44\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.386533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.386438    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:19.722528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.722502    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"]
Oct 30 03:21:19.723187 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:21:19.723164    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79415bef_b337_4492_9ab8_3c3081ce7574.slice/crio-1e0e262c3e15f6b97580b91a03d1124550684d4c2910bfdd54e67d6acac9e63b WatchSource:0}: Error finding container 1e0e262c3e15f6b97580b91a03d1124550684d4c2910bfdd54e67d6acac9e63b: Status 404 returned error can't find the container with id 1e0e262c3e15f6b97580b91a03d1124550684d4c2910bfdd54e67d6acac9e63b
Oct 30 03:21:19.726160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.726129    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:19.726235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.726191    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:19.726235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:19.726219    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:20.276976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.276934    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" event={"ID":"79415bef-b337-4492-9ab8-3c3081ce7574","Type":"ContainerStarted","Data":"2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b"}
Oct 30 03:21:20.277474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.277042    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" event={"ID":"79415bef-b337-4492-9ab8-3c3081ce7574","Type":"ContainerStarted","Data":"1e0e262c3e15f6b97580b91a03d1124550684d4c2910bfdd54e67d6acac9e63b"}
Oct 30 03:21:20.339771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.339716    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podStartSLOduration=1.339702024 podStartE2EDuration="1.339702024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:20.336879785 +0000 UTC m=+1108.765769207" watchObservedRunningTime="2026-10-30 03:21:20.339702024 +0000 UTC m=+1108.768591441"
Oct 30 03:21:20.386606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.386570    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:20.388038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.388010    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:20.388198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.388072    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:20.927053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.927014    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"]
Oct 30 03:21:20.927465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:20.927412    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" containerID="cri-o://9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96" gracePeriod=30
Oct 30 03:21:21.389959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:21.389278    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:21.389959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:21.389361    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:22.398447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:22.392668    4877 patch_prober.go:29] interesting pod/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:21:22.398447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:22.392733    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:21:22.398447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:22.393233    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:22.398447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:22.393274    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:23.413864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:23.413767    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:23.413864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:23.413825    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:24.403698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.403662    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:24.403991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.403963    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:24.913072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.913035    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"]
Oct 30 03:21:24.918051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.918025    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:24.933104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.932994    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:21:24.936829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.936804    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"]
Oct 30 03:21:24.948450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:24.948394    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:21:25.046549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.046518    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlwk\" (UniqueName: \"kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.046720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.046571    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.046720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.046691    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.148384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.148348    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.148577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.148406    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlwk\" (UniqueName: \"kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.148577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.148477    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.149186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.149152    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.150741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.150719    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.167363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.167305    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlwk\" (UniqueName: \"kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk\") pod \"maas-api-revoke-sub-3873930d-gjp5j\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.234853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.234818    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:25.412545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.412004    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:25.412545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.412058    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:25.623015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:25.616882    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"]
Oct 30 03:21:26.293919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.293882    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:21:26.372106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.371960    4877 generic.go:356] "Generic (PLEG): container finished" podID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerID="9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96" exitCode=0
Oct 30 03:21:26.372294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.372121    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" event={"ID":"624f9351-9f13-49d9-9d99-aee4bf695eed","Type":"ContainerDied","Data":"9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96"}
Oct 30 03:21:26.372294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.372174    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx" event={"ID":"624f9351-9f13-49d9-9d99-aee4bf695eed","Type":"ContainerDied","Data":"efd88d3518513251fb1858c93a1d414c551ee6648459c7d72ddddb5fc2cbfb84"}
Oct 30 03:21:26.372294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.372206    4877 scope.go:122] "RemoveContainer" containerID="9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96"
Oct 30 03:21:26.372441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.372371    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"
Oct 30 03:21:26.373148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373122    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373192    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373282 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373267    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373346    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373392    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373443    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373499    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373664    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.373727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.373706    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token\") pod \"624f9351-9f13-49d9-9d99-aee4bf695eed\" (UID: \"624f9351-9f13-49d9-9d99-aee4bf695eed\") "
Oct 30 03:21:26.376050 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.376011    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:26.376256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.376230    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:26.376513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.376485    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:26.377538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.377508    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:26.378099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.378076    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:26.394986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.380414    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:26.401469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.397760    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:21:26.401469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.398440    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:26.401469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.398488    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:26.401469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.400733    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:26.404902 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.404862    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk" pod "624f9351-9f13-49d9-9d99-aee4bf695eed" (UID: "624f9351-9f13-49d9-9d99-aee4bf695eed"). InnerVolumeSpecName "kube-api-access-665fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:26.408745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.407679    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j" event={"ID":"9bbd4589-78f2-4987-8955-4c20fab7c060","Type":"ContainerStarted","Data":"b4fb38a8a1f7341d6f04fd6c852124bc5c66ea2a079f02cfc08f18d1193d4a6b"}
Oct 30 03:21:26.408745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.407738    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j" event={"ID":"9bbd4589-78f2-4987-8955-4c20fab7c060","Type":"ContainerStarted","Data":"cad021adffa755ed3122a196ce12620fe5f63fe3bd76fb0cd363234437e4f618"}
Oct 30 03:21:26.435829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.435768    4877 scope.go:122] "RemoveContainer" containerID="9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96"
Oct 30 03:21:26.437476 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:26.436091    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96\": container with ID starting with 9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96 not found: ID does not exist" containerID="9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96"
Oct 30 03:21:26.437476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.436134    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96"} err="failed to get container status \"9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96\": rpc error: code = NotFound desc = could not find container \"9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96\": container with ID starting with 9583c157a1156ee747409684caaf91094b83c895d79bdf669441e3a579511b96 not found: ID does not exist"
Oct 30 03:21:26.455489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.455440    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j" podStartSLOduration=2.45541149 podStartE2EDuration="2.45541149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:26.451356759 +0000 UTC m=+1114.880246176" watchObservedRunningTime="2026-10-30 03:21:26.45541149 +0000 UTC m=+1114.884300920"
Oct 30 03:21:26.475510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475477    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475504    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665fk\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-kube-api-access-665fk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475519    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475535    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/624f9351-9f13-49d9-9d99-aee4bf695eed-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475549    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475561    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475573    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475587    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.475704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.475602    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/624f9351-9f13-49d9-9d99-aee4bf695eed-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:26.719080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.719045    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"]
Oct 30 03:21:26.724373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:26.724345    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cc0859-openshift-default-d7ddc779-2bkgx"]
Oct 30 03:21:27.387573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:27.387536    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 03:21:27.387986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:27.387619    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 03:21:27.429486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:27.429357    4877 generic.go:356] "Generic (PLEG): container finished" podID="9bbd4589-78f2-4987-8955-4c20fab7c060" containerID="b4fb38a8a1f7341d6f04fd6c852124bc5c66ea2a079f02cfc08f18d1193d4a6b" exitCode=0
Oct 30 03:21:27.429486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:27.429450    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j" event={"ID":"9bbd4589-78f2-4987-8955-4c20fab7c060","Type":"ContainerDied","Data":"b4fb38a8a1f7341d6f04fd6c852124bc5c66ea2a079f02cfc08f18d1193d4a6b"}
Oct 30 03:21:27.701292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:27.701204    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" path="/var/lib/kubelet/pods/624f9351-9f13-49d9-9d99-aee4bf695eed/volumes"
Oct 30 03:21:28.654943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.654860    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:28.814317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.814283    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk\" (UniqueName: \"kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk\") pod \"9bbd4589-78f2-4987-8955-4c20fab7c060\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") "
Oct 30 03:21:28.814545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.814335    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca\") pod \"9bbd4589-78f2-4987-8955-4c20fab7c060\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") "
Oct 30 03:21:28.814545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.814483    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca\") pod \"9bbd4589-78f2-4987-8955-4c20fab7c060\" (UID: \"9bbd4589-78f2-4987-8955-4c20fab7c060\") "
Oct 30 03:21:28.814841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.814816    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca" pod "9bbd4589-78f2-4987-8955-4c20fab7c060" (UID: "9bbd4589-78f2-4987-8955-4c20fab7c060"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:28.815179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.815133    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:28.817358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.816690    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk" pod "9bbd4589-78f2-4987-8955-4c20fab7c060" (UID: "9bbd4589-78f2-4987-8955-4c20fab7c060"). InnerVolumeSpecName "kube-api-access-rrlwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:28.817358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.816962    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca" pod "9bbd4589-78f2-4987-8955-4c20fab7c060" (UID: "9bbd4589-78f2-4987-8955-4c20fab7c060"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:28.817880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.817813    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:28.818969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.818951    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:28.819792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.819774    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:21:28.916445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.916282    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrlwk\" (UniqueName: \"kubernetes.io/projected/9bbd4589-78f2-4987-8955-4c20fab7c060-kube-api-access-rrlwk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:28.916445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:28.916317    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbd4589-78f2-4987-8955-4c20fab7c060-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:29.448269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:29.448236    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"
Oct 30 03:21:29.448539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:29.448266    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j" event={"ID":"9bbd4589-78f2-4987-8955-4c20fab7c060","Type":"ContainerDied","Data":"cad021adffa755ed3122a196ce12620fe5f63fe3bd76fb0cd363234437e4f618"}
Oct 30 03:21:29.448539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:29.448325    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad021adffa755ed3122a196ce12620fe5f63fe3bd76fb0cd363234437e4f618"
Oct 30 03:21:33.413744 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.413666    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"]
Oct 30 03:21:33.414637 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.414617    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9bbd4589-78f2-4987-8955-4c20fab7c060" containerName="revoke-keys"
Oct 30 03:21:33.414699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.414639    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbd4589-78f2-4987-8955-4c20fab7c060" containerName="revoke-keys"
Oct 30 03:21:33.414699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.414682    4877 cpu_manager.go:414] "containerMap: removing container" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy"
Oct 30 03:21:33.414699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.414691    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy"
Oct 30 03:21:33.414873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.414861    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbd4589-78f2-4987-8955-4c20fab7c060" containerName="revoke-keys"
Oct 30 03:21:33.414927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.414879    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="624f9351-9f13-49d9-9d99-aee4bf695eed" containerName="istio-proxy"
Oct 30 03:21:33.420662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.420635    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.442113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.442087    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"]
Oct 30 03:21:33.448473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.448449    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d6c536-openshift-default-dockercfg-ttszv\""
Oct 30 03:21:33.482521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482482    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.482647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482533    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkrv\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.482706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482664    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.482755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482709    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.482814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482792    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.482857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482832    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.482964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482948    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.483010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.482995    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.483078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.483063    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584364    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584407    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584478    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584634    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkrv\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584733    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584803    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584886    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584925    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.584967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584960    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.585123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.584986    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.585123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.585007    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.585224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.585203    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.585368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.585348    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.588143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.588113    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.594154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.594125    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.622202 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.622164    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkrv\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.624807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.624776    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token\") pod \"e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") " pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:33.749530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:33.744550    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:34.099262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.099226    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"]
Oct 30 03:21:34.100651 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:21:34.100620    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3b31c2_a4df_4aed_b955_0639253f104c.slice/crio-0ed62f70bd2f315cd31220ee9ed62e330f0065d6a8619a27b0a3e27d66dfec3f WatchSource:0}: Error finding container 0ed62f70bd2f315cd31220ee9ed62e330f0065d6a8619a27b0a3e27d66dfec3f: Status 404 returned error can't find the container with id 0ed62f70bd2f315cd31220ee9ed62e330f0065d6a8619a27b0a3e27d66dfec3f
Oct 30 03:21:34.103056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.103023    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:34.103167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.103089    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:34.103167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.103117    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:34.489794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.489694    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" event={"ID":"4e3b31c2-a4df-4aed-b955-0639253f104c","Type":"ContainerStarted","Data":"0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"}
Oct 30 03:21:34.490211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.489798    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" event={"ID":"4e3b31c2-a4df-4aed-b955-0639253f104c","Type":"ContainerStarted","Data":"0ed62f70bd2f315cd31220ee9ed62e330f0065d6a8619a27b0a3e27d66dfec3f"}
Oct 30 03:21:34.539297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.539241    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podStartSLOduration=1.539227392 podStartE2EDuration="1.539227392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:34.53556032 +0000 UTC m=+1122.964449739" watchObservedRunningTime="2026-10-30 03:21:34.539227392 +0000 UTC m=+1122.968116809"
Oct 30 03:21:34.745970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:34.745895    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:35.753530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:35.753494    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:36.528476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:36.522038    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:36.528476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:36.527594    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:21:40.203543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.203504    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"]
Oct 30 03:21:40.207847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.207822    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.228236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.228207    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"]
Oct 30 03:21:40.261526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.261475    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6c536-dockercfg-4wmlr\""
Oct 30 03:21:40.270714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.270668    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6c536\""
Oct 30 03:21:40.323448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.305250    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.323448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.305311    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxqt\" (UniqueName: \"kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt\") pod \"payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.425088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.425054    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.425265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.425106    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxqt\" (UniqueName: \"kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt\") pod \"payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.426063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.426033    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.445038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.445001    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxqt\" (UniqueName: \"kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt\") pod \"payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.524334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.524294    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:40.837536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:40.837508    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"]
Oct 30 03:21:40.839703 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:21:40.839672    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca409c7a_d071_41bb_a21c_a828b8dfe651.slice/crio-7af678d6a6f37a9b129e6c406de4d2d9ba232b170a3334957641a79906b848e2 WatchSource:0}: Error finding container 7af678d6a6f37a9b129e6c406de4d2d9ba232b170a3334957641a79906b848e2: Status 404 returned error can't find the container with id 7af678d6a6f37a9b129e6c406de4d2d9ba232b170a3334957641a79906b848e2
Oct 30 03:21:41.647479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:41.647413    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" event={"ID":"ca409c7a-d071-41bb-a21c-a828b8dfe651","Type":"ContainerStarted","Data":"a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e"}
Oct 30 03:21:41.647982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:41.647536    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" event={"ID":"ca409c7a-d071-41bb-a21c-a828b8dfe651","Type":"ContainerStarted","Data":"7af678d6a6f37a9b129e6c406de4d2d9ba232b170a3334957641a79906b848e2"}
Oct 30 03:21:41.647982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:41.647567    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:41.686810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:41.686749    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" podStartSLOduration=1.686732581 podStartE2EDuration="1.686732581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:41.685915611 +0000 UTC m=+1130.114805032" watchObservedRunningTime="2026-10-30 03:21:41.686732581 +0000 UTC m=+1130.115621999"
Oct 30 03:21:46.230376 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:46.230330    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:21:46.614518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:46.614480    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"]
Oct 30 03:21:46.644474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:46.644439    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"]
Oct 30 03:21:51.615564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.615489    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9" podUID="a717ddf0-ac5b-453c-8ac8-6ba1813d1752" containerName="payload-processing" containerID="cri-o://d12a3cb87809a052947c83acd69d933df35be05d6d313f643f6e17f722f18d03" gracePeriod=25
Oct 30 03:21:51.644979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.644941    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz" podUID="35159481-21fc-4ce5-9c49-9050fe128365" containerName="payload-pre-processing" containerID="cri-o://fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e" gracePeriod=25
Oct 30 03:21:51.744392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.744361    4877 generic.go:356] "Generic (PLEG): container finished" podID="a717ddf0-ac5b-453c-8ac8-6ba1813d1752" containerID="d12a3cb87809a052947c83acd69d933df35be05d6d313f643f6e17f722f18d03" exitCode=0
Oct 30 03:21:51.744565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.744452    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9" event={"ID":"a717ddf0-ac5b-453c-8ac8-6ba1813d1752","Type":"ContainerDied","Data":"d12a3cb87809a052947c83acd69d933df35be05d6d313f643f6e17f722f18d03"}
Oct 30 03:21:51.928663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.928636    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:21:51.962055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.962028    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:21:51.977971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.977940    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p\" (UniqueName: \"kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p\") pod \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") "
Oct 30 03:21:51.978101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.977989    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume\") pod \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\" (UID: \"a717ddf0-ac5b-453c-8ac8-6ba1813d1752\") "
Oct 30 03:21:51.978806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.978777    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume" pod "a717ddf0-ac5b-453c-8ac8-6ba1813d1752" (UID: "a717ddf0-ac5b-453c-8ac8-6ba1813d1752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:51.980925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:51.980896    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p" pod "a717ddf0-ac5b-453c-8ac8-6ba1813d1752" (UID: "a717ddf0-ac5b-453c-8ac8-6ba1813d1752"). InnerVolumeSpecName "kube-api-access-jqf7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:52.079779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.079749    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume\") pod \"35159481-21fc-4ce5-9c49-9050fe128365\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") "
Oct 30 03:21:52.079989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.079794    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv\" (UniqueName: \"kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv\") pod \"35159481-21fc-4ce5-9c49-9050fe128365\" (UID: \"35159481-21fc-4ce5-9c49-9050fe128365\") "
Oct 30 03:21:52.080158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.080133    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume" pod "35159481-21fc-4ce5-9c49-9050fe128365" (UID: "35159481-21fc-4ce5-9c49-9050fe128365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:52.080290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.080277    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqf7p\" (UniqueName: \"kubernetes.io/projected/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-kube-api-access-jqf7p\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:52.080330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.080293    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a717ddf0-ac5b-453c-8ac8-6ba1813d1752-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:52.080330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.080302    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35159481-21fc-4ce5-9c49-9050fe128365-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:52.081915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.081892    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv" pod "35159481-21fc-4ce5-9c49-9050fe128365" (UID: "35159481-21fc-4ce5-9c49-9050fe128365"). InnerVolumeSpecName "kube-api-access-6bbhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:52.182202 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.182113    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbhv\" (UniqueName: \"kubernetes.io/projected/35159481-21fc-4ce5-9c49-9050fe128365-kube-api-access-6bbhv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:21:52.660070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.660042    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:21:52.756014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.755978    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"
Oct 30 03:21:52.756608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.756545    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9" event={"ID":"a717ddf0-ac5b-453c-8ac8-6ba1813d1752","Type":"ContainerDied","Data":"729f562c48e9b10b34a6c5b7791fb32914fe60c1122ad614d06dddf0a290b9c2"}
Oct 30 03:21:52.756748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.756620    4877 scope.go:122] "RemoveContainer" containerID="d12a3cb87809a052947c83acd69d933df35be05d6d313f643f6e17f722f18d03"
Oct 30 03:21:52.772780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.772744    4877 generic.go:356] "Generic (PLEG): container finished" podID="35159481-21fc-4ce5-9c49-9050fe128365" containerID="fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e" exitCode=0
Oct 30 03:21:52.772928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.772863    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"
Oct 30 03:21:52.772928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.772912    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz" event={"ID":"35159481-21fc-4ce5-9c49-9050fe128365","Type":"ContainerDied","Data":"fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e"}
Oct 30 03:21:52.773031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.772964    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz" event={"ID":"35159481-21fc-4ce5-9c49-9050fe128365","Type":"ContainerDied","Data":"7e9a40bdb4cd6fc6094500c3cd271403045e2c80c9eb116595c5ec35271c30bd"}
Oct 30 03:21:52.782023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.781998    4877 scope.go:122] "RemoveContainer" containerID="fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e"
Oct 30 03:21:52.796414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.796388    4877 scope.go:122] "RemoveContainer" containerID="fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e"
Oct 30 03:21:52.796827 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:21:52.796803    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e\": container with ID starting with fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e not found: ID does not exist" containerID="fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e"
Oct 30 03:21:52.796933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.796839    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e"} err="failed to get container status \"fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e\": rpc error: code = NotFound desc = could not find container \"fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e\": container with ID starting with fa936518a6f0a88b77760f6c940b10f87fc834632cc536e6b4c10287e3886a3e not found: ID does not exist"
Oct 30 03:21:52.810247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.810211    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"]
Oct 30 03:21:52.816523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.816495    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3b93b5-f4d8d5c5d-hn5w9"]
Oct 30 03:21:52.849792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.849757    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"]
Oct 30 03:21:52.855013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:52.854963    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3b93b5-66b74d8555-zvmrz"]
Oct 30 03:21:53.702640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:53.702603    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35159481-21fc-4ce5-9c49-9050fe128365" path="/var/lib/kubelet/pods/35159481-21fc-4ce5-9c49-9050fe128365/volumes"
Oct 30 03:21:53.703223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:53.703199    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a717ddf0-ac5b-453c-8ac8-6ba1813d1752" path="/var/lib/kubelet/pods/a717ddf0-ac5b-453c-8ac8-6ba1813d1752/volumes"
Oct 30 03:21:54.012339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:54.012260    4877 scope.go:122] "RemoveContainer" containerID="78b95c0647562ac7bdc02511a74473277b1832a9c300a8b8e4547a9420d54a37"
Oct 30 03:21:54.023983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:54.023959    4877 scope.go:122] "RemoveContainer" containerID="4d699ca5e010fb97969d4ae8d4b4c6f41d44c247969e14c7e89f7e109717edde"
Oct 30 03:21:59.325588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.325548    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"]
Oct 30 03:21:59.326477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.326459    4877 cpu_manager.go:414] "containerMap: removing container" podUID="35159481-21fc-4ce5-9c49-9050fe128365" containerName="payload-pre-processing"
Oct 30 03:21:59.326477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.326477    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35159481-21fc-4ce5-9c49-9050fe128365" containerName="payload-pre-processing"
Oct 30 03:21:59.326623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.326543    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a717ddf0-ac5b-453c-8ac8-6ba1813d1752" containerName="payload-processing"
Oct 30 03:21:59.326623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.326551    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a717ddf0-ac5b-453c-8ac8-6ba1813d1752" containerName="payload-processing"
Oct 30 03:21:59.326776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.326759    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="35159481-21fc-4ce5-9c49-9050fe128365" containerName="payload-pre-processing"
Oct 30 03:21:59.326831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.326787    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a717ddf0-ac5b-453c-8ac8-6ba1813d1752" containerName="payload-processing"
Oct 30 03:21:59.332067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.332047    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.348114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.348089    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-04fdf3-openshift-default-dockercfg-dd4bf\""
Oct 30 03:21:59.366121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.366088    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"]
Oct 30 03:21:59.377269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377230    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377349    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377376    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377399    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377531    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377591    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzld\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377610    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377675    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.377848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.377823    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.487827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.485452    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.487827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.485555    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.487827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.485616    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.487827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.486975    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.487827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.487693    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.490267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.489806    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.490513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.490490    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.491086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.491058    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzld\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.491226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.491212    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.491347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.491332    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.491575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.491557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.491799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.491783    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.492015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.491801    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.492124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.492006    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.498029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.498000    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.517347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.507076    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.517347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.516946    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzld\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.526999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.526971    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token\") pod \"e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") " pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.648989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.648904    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:21:59.979370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.979334    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"]
Oct 30 03:21:59.979539 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:21:59.979517    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075ae435_693b_406e_ad16_3d253430068b.slice/crio-229a23f720a34f43b8ebf7c6dd9d0fda6a0334d795b0cb6b782ddf6be251104e WatchSource:0}: Error finding container 229a23f720a34f43b8ebf7c6dd9d0fda6a0334d795b0cb6b782ddf6be251104e: Status 404 returned error can't find the container with id 229a23f720a34f43b8ebf7c6dd9d0fda6a0334d795b0cb6b782ddf6be251104e
Oct 30 03:21:59.982277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.982237    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:59.982400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.982318    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:21:59.982400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:21:59.982365    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:00.841869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:00.841830    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" event={"ID":"075ae435-693b-406e-ad16-3d253430068b","Type":"ContainerStarted","Data":"b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"}
Oct 30 03:22:00.842265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:00.841885    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" event={"ID":"075ae435-693b-406e-ad16-3d253430068b","Type":"ContainerStarted","Data":"229a23f720a34f43b8ebf7c6dd9d0fda6a0334d795b0cb6b782ddf6be251104e"}
Oct 30 03:22:00.898849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:00.898787    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podStartSLOduration=1.898768995 podStartE2EDuration="1.898768995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:00.896075216 +0000 UTC m=+1149.324964661" watchObservedRunningTime="2026-10-30 03:22:00.898768995 +0000 UTC m=+1149.327658414"
Oct 30 03:22:01.649746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:01.649709    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:22:01.655413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:01.655387    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:22:01.848852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:01.848815    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:22:01.849772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:01.849752    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:22:05.321068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.321022    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"]
Oct 30 03:22:05.327591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.327564    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.355726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.355688    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"]
Oct 30 03:22:05.472446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.471234    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04fdf3-dockercfg-vbk5m\""
Oct 30 03:22:05.487476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.486827    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04fdf3\""
Oct 30 03:22:05.490569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.488826    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.490569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.488946    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8t2j\" (UniqueName: \"kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.602502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.601608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8t2j\" (UniqueName: \"kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.602502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.601815    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.609736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.602852    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.622689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.622640    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8t2j\" (UniqueName: \"kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:05.650838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:05.650809    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:06.016478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:06.016056    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"]
Oct 30 03:22:06.029464 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:06.029410    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947cd9d9_bafe_4ba8_8ca1_218163942042.slice/crio-f003c1414fb1b94e98f2907ec85da0c6c03ec2118f89fa17b03dd3f2b834c8b0 WatchSource:0}: Error finding container f003c1414fb1b94e98f2907ec85da0c6c03ec2118f89fa17b03dd3f2b834c8b0: Status 404 returned error can't find the container with id f003c1414fb1b94e98f2907ec85da0c6c03ec2118f89fa17b03dd3f2b834c8b0
Oct 30 03:22:06.928724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:06.928677    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp" event={"ID":"947cd9d9-bafe-4ba8-8ca1-218163942042","Type":"ContainerStarted","Data":"5aba779c3a5ca1b707ae52c814996f4c0e827b4c2af533ff4535378cc9684c27"}
Oct 30 03:22:06.929105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:06.928758    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp" event={"ID":"947cd9d9-bafe-4ba8-8ca1-218163942042","Type":"ContainerStarted","Data":"f003c1414fb1b94e98f2907ec85da0c6c03ec2118f89fa17b03dd3f2b834c8b0"}
Oct 30 03:22:06.929277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:06.929256    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:06.968836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:06.968737    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp" podStartSLOduration=1.96872293 podStartE2EDuration="1.96872293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:06.967999384 +0000 UTC m=+1155.396888837" watchObservedRunningTime="2026-10-30 03:22:06.96872293 +0000 UTC m=+1155.397612347"
Oct 30 03:22:08.042633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:08.042238    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"]
Oct 30 03:22:08.042633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:08.042507    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" containerID="cri-o://448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c" gracePeriod=30
Oct 30 03:22:08.528060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:08.528026    4877 patch_prober.go:29] interesting pod/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:22:08.528226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:08.528086    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:22:12.984138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:12.984104    4877 generic.go:356] "Generic (PLEG): container finished" podID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerID="0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c" exitCode=0
Oct 30 03:22:12.984687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:12.984178    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" event={"ID":"4e3b31c2-a4df-4aed-b955-0639253f104c","Type":"ContainerDied","Data":"0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"}
Oct 30 03:22:12.984753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:12.984737    4877 scope.go:122] "RemoveContainer" containerID="0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"
Oct 30 03:22:12.986280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:12.986249    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:12.986400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:12.986362    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:12.986500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:12.986411    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:13.343444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.343389    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:22:13.409280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409243    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409379    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409452    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409485    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409516    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409549    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409578    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409651    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.409831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.409729    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token\") pod \"8b58b090-24eb-4272-bd9d-ddb332828f3e\" (UID: \"8b58b090-24eb-4272-bd9d-ddb332828f3e\") "
Oct 30 03:22:13.411231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.410762    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:13.411231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.410850    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:13.411231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.410987    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:13.411569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.411529    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:13.411828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.411804    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:13.412994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.412941    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:13.413890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.413862    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:13.414581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.414551    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "kube-api-access-9hn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:13.415147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.415119    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo" pod "8b58b090-24eb-4272-bd9d-ddb332828f3e" (UID: "8b58b090-24eb-4272-bd9d-ddb332828f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511892    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511921    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hn9p\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-kube-api-access-9hn9p\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511931    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511940    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511949    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511957    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511964    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511972    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b58b090-24eb-4272-bd9d-ddb332828f3e-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:13.511971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:13.511979    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b58b090-24eb-4272-bd9d-ddb332828f3e-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:14.007723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.006462    4877 generic.go:356] "Generic (PLEG): container finished" podID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerID="448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c" exitCode=0
Oct 30 03:22:14.007723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.006593    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" event={"ID":"8b58b090-24eb-4272-bd9d-ddb332828f3e","Type":"ContainerDied","Data":"448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c"}
Oct 30 03:22:14.007723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.006645    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d" event={"ID":"8b58b090-24eb-4272-bd9d-ddb332828f3e","Type":"ContainerDied","Data":"59a336b3f39a0b382c5d8525d4ea3c313108ac952eb6ff6047dc3262553c2a87"}
Oct 30 03:22:14.007723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.006680    4877 scope.go:122] "RemoveContainer" containerID="448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c"
Oct 30 03:22:14.007723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.006857    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"
Oct 30 03:22:14.011030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.010999    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" event={"ID":"4e3b31c2-a4df-4aed-b955-0639253f104c","Type":"ContainerStarted","Data":"c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe"}
Oct 30 03:22:14.018760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.018734    4877 scope.go:122] "RemoveContainer" containerID="5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"
Oct 30 03:22:14.028489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.028465    4877 scope.go:122] "RemoveContainer" containerID="448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c"
Oct 30 03:22:14.028773 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:14.028753    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c\": container with ID starting with 448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c not found: ID does not exist" containerID="448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c"
Oct 30 03:22:14.028845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.028782    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c"} err="failed to get container status \"448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c\": rpc error: code = NotFound desc = could not find container \"448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c\": container with ID starting with 448cc35919ce698b2f51136cdc912568731efb63f8ba9b571441d6d735c3f08c not found: ID does not exist"
Oct 30 03:22:14.028845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.028801    4877 scope.go:122] "RemoveContainer" containerID="5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"
Oct 30 03:22:14.029053 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:14.029034    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf\": container with ID starting with 5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf not found: ID does not exist" containerID="5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"
Oct 30 03:22:14.029118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.029060    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf"} err="failed to get container status \"5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf\": rpc error: code = NotFound desc = could not find container \"5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf\": container with ID starting with 5f4ffb0265b5c1fcc29777e7f0062c9582ebe0788c9debf0e35f8aa66ada3bcf not found: ID does not exist"
Oct 30 03:22:14.058632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.058598    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"]
Oct 30 03:22:14.070488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.070460    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3b93b5-openshift-default-d878cd889-gtt8d"]
Oct 30 03:22:14.745380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.745347    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:22:14.745584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.745544    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 03:22:14.745636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:14.745589    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 03:22:15.606072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.606029    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"]
Oct 30 03:22:15.635922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.635888    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"]
Oct 30 03:22:15.730969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.730933    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" path="/var/lib/kubelet/pods/8b58b090-24eb-4272-bd9d-ddb332828f3e/volumes"
Oct 30 03:22:15.745874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.745813    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 03:22:15.745874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.745868    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 03:22:15.917407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.916615    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"]
Oct 30 03:22:15.920635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:15.920587    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"]
Oct 30 03:22:16.325973 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:16.325932    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:22:16.745500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:16.745402    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 03:22:16.745500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:16.745470    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 03:22:17.512338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:17.512301    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"]
Oct 30 03:22:17.751463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:17.750947    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 03:22:17.751463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:17.750992    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 03:22:17.940316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:17.940278    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:18.746236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:18.746193    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 03:22:18.746438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:18.746251    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 03:22:19.750267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:19.750238    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:22:19.750735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:19.750721    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:22:19.751343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:19.751322    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:22:20.607043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:20.606992    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c" podUID="1a6038ad-db04-4cf3-88d1-874a87a7776d" containerName="payload-processing" containerID="cri-o://1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82" gracePeriod=25
Oct 30 03:22:20.637203 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:20.637143    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf" podUID="fde1591a-2b1e-4c45-b5a5-06f0145e00f6" containerName="payload-pre-processing" containerID="cri-o://6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180" gracePeriod=25
Oct 30 03:22:20.917739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:20.917699    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" podUID="bb73ac0a-d179-41ed-a82e-f182b7167e60" containerName="payload-processing" containerID="cri-o://deb7292adfaeee5c62ac4b4884039760ce5f622e4dc5bed2a18c379f6e0e32db" gracePeriod=25
Oct 30 03:22:20.922185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:20.922147    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" podUID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" containerName="payload-pre-processing" containerID="cri-o://7cd00a0c8c42c1a73257d0c6890d63df6336227d2af6866eb6ebcca58a121ff0" gracePeriod=25
Oct 30 03:22:20.931074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:20.931049    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:22:20.980081 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:20.980053    4877 logging.go:55] [core] [Channel #2073 SubChannel #2075]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.117:50052", ServerName: "10.133.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.117:50052: connect: connection refused"
Oct 30 03:22:20.980235 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:20.980064    4877 logging.go:55] [core] [Channel #2072 SubChannel #2074]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.116:50052", ServerName: "10.133.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.116:50052: connect: connection refused"
Oct 30 03:22:21.037461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.037408    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:22:21.055566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.055532    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2\" (UniqueName: \"kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2\") pod \"1a6038ad-db04-4cf3-88d1-874a87a7776d\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") "
Oct 30 03:22:21.055747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.055735    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume\") pod \"1a6038ad-db04-4cf3-88d1-874a87a7776d\" (UID: \"1a6038ad-db04-4cf3-88d1-874a87a7776d\") "
Oct 30 03:22:21.056204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.056174    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume" pod "1a6038ad-db04-4cf3-88d1-874a87a7776d" (UID: "1a6038ad-db04-4cf3-88d1-874a87a7776d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:21.058165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.058135    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2" pod "1a6038ad-db04-4cf3-88d1-874a87a7776d" (UID: "1a6038ad-db04-4cf3-88d1-874a87a7776d"). InnerVolumeSpecName "kube-api-access-4w4c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:21.070939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.070910    4877 generic.go:356] "Generic (PLEG): container finished" podID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" containerID="7cd00a0c8c42c1a73257d0c6890d63df6336227d2af6866eb6ebcca58a121ff0" exitCode=0
Oct 30 03:22:21.071111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.071065    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" event={"ID":"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8","Type":"ContainerDied","Data":"7cd00a0c8c42c1a73257d0c6890d63df6336227d2af6866eb6ebcca58a121ff0"}
Oct 30 03:22:21.072590 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.072566    4877 generic.go:356] "Generic (PLEG): container finished" podID="fde1591a-2b1e-4c45-b5a5-06f0145e00f6" containerID="6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180" exitCode=0
Oct 30 03:22:21.072682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.072672    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"
Oct 30 03:22:21.072800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.072788    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf" event={"ID":"fde1591a-2b1e-4c45-b5a5-06f0145e00f6","Type":"ContainerDied","Data":"6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180"}
Oct 30 03:22:21.072836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.072816    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf" event={"ID":"fde1591a-2b1e-4c45-b5a5-06f0145e00f6","Type":"ContainerDied","Data":"f58cc12ca61e41751f038ce3960a56c922c41d5f28598ac58b2366b10cfc1000"}
Oct 30 03:22:21.072880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.072836    4877 scope.go:122] "RemoveContainer" containerID="6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180"
Oct 30 03:22:21.084665 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.084502    4877 generic.go:356] "Generic (PLEG): container finished" podID="bb73ac0a-d179-41ed-a82e-f182b7167e60" containerID="deb7292adfaeee5c62ac4b4884039760ce5f622e4dc5bed2a18c379f6e0e32db" exitCode=0
Oct 30 03:22:21.084665 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.084596    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" event={"ID":"bb73ac0a-d179-41ed-a82e-f182b7167e60","Type":"ContainerDied","Data":"deb7292adfaeee5c62ac4b4884039760ce5f622e4dc5bed2a18c379f6e0e32db"}
Oct 30 03:22:21.089576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.089545    4877 generic.go:356] "Generic (PLEG): container finished" podID="1a6038ad-db04-4cf3-88d1-874a87a7776d" containerID="1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82" exitCode=0
Oct 30 03:22:21.092313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.089621    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c" event={"ID":"1a6038ad-db04-4cf3-88d1-874a87a7776d","Type":"ContainerDied","Data":"1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82"}
Oct 30 03:22:21.092313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.090733    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c" event={"ID":"1a6038ad-db04-4cf3-88d1-874a87a7776d","Type":"ContainerDied","Data":"7eac89d8d3572b8396286d96f8bf7ca426de2e25beefd824662fed8440e3978f"}
Oct 30 03:22:21.092313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.089654    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"
Oct 30 03:22:21.092313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.092201    4877 scope.go:122] "RemoveContainer" containerID="6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180"
Oct 30 03:22:21.104402 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:21.092744    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180\": container with ID starting with 6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180 not found: ID does not exist" containerID="6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180"
Oct 30 03:22:21.104402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.092778    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180"} err="failed to get container status \"6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180\": rpc error: code = NotFound desc = could not find container \"6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180\": container with ID starting with 6dc9cbcff98664106145164387812732e531c5866fe2304674e27d6036697180 not found: ID does not exist"
Oct 30 03:22:21.104402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.092801    4877 scope.go:122] "RemoveContainer" containerID="1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82"
Oct 30 03:22:21.138175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.138157    4877 scope.go:122] "RemoveContainer" containerID="1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82"
Oct 30 03:22:21.138637 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:21.138610    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82\": container with ID starting with 1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82 not found: ID does not exist" containerID="1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82"
Oct 30 03:22:21.138722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.138645    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82"} err="failed to get container status \"1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82\": rpc error: code = NotFound desc = could not find container \"1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82\": container with ID starting with 1e6a755c449e1ec124a83da64cf7251a1d770b23a5c5d6b2ded2c16231fbbc82 not found: ID does not exist"
Oct 30 03:22:21.147728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.147704    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"]
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.157530    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd\" (UniqueName: \"kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd\") pod \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") "
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.158303    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume\") pod \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\" (UID: \"fde1591a-2b1e-4c45-b5a5-06f0145e00f6\") "
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.158815    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume" pod "fde1591a-2b1e-4c45-b5a5-06f0145e00f6" (UID: "fde1591a-2b1e-4c45-b5a5-06f0145e00f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.159008    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6038ad-db04-4cf3-88d1-874a87a7776d-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.159028    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.159043    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4c2\" (UniqueName: \"kubernetes.io/projected/1a6038ad-db04-4cf3-88d1-874a87a7776d-kube-api-access-4w4c2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.163617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.162342    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd" pod "fde1591a-2b1e-4c45-b5a5-06f0145e00f6" (UID: "fde1591a-2b1e-4c45-b5a5-06f0145e00f6"). InnerVolumeSpecName "kube-api-access-m4tnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:21.165653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.165297    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b0d5-7978d5d8c8-6gc5c"]
Oct 30 03:22:21.239873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.239842    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:22:21.241924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.241908    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:22:21.260276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.260245    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4tnd\" (UniqueName: \"kubernetes.io/projected/fde1591a-2b1e-4c45-b5a5-06f0145e00f6-kube-api-access-m4tnd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.361089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.361056    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume\") pod \"bb73ac0a-d179-41ed-a82e-f182b7167e60\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") "
Oct 30 03:22:21.361270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.361152    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv\" (UniqueName: \"kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv\") pod \"bb73ac0a-d179-41ed-a82e-f182b7167e60\" (UID: \"bb73ac0a-d179-41ed-a82e-f182b7167e60\") "
Oct 30 03:22:21.361270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.361204    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr\" (UniqueName: \"kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr\") pod \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") "
Oct 30 03:22:21.361367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.361301    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume\") pod \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\" (UID: \"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8\") "
Oct 30 03:22:21.361829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.361721    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume" pod "3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" (UID: "3a5a4c67-d88e-4bda-9fa2-ea6393f074b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:21.361961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.361923    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.362088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.362066    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume" pod "bb73ac0a-d179-41ed-a82e-f182b7167e60" (UID: "bb73ac0a-d179-41ed-a82e-f182b7167e60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:21.363520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.363495    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv" pod "bb73ac0a-d179-41ed-a82e-f182b7167e60" (UID: "bb73ac0a-d179-41ed-a82e-f182b7167e60"). InnerVolumeSpecName "kube-api-access-rcrhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:21.363597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.363535    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr" pod "3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" (UID: "3a5a4c67-d88e-4bda-9fa2-ea6393f074b8"). InnerVolumeSpecName "kube-api-access-qnnvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:21.417386 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.417293    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"]
Oct 30 03:22:21.421100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.421069    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b0d5-76dcbfb975-d84xf"]
Oct 30 03:22:21.463311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.463277    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb73ac0a-d179-41ed-a82e-f182b7167e60-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.463311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.463304    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrhv\" (UniqueName: \"kubernetes.io/projected/bb73ac0a-d179-41ed-a82e-f182b7167e60-kube-api-access-rcrhv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.463311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.463316    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnnvr\" (UniqueName: \"kubernetes.io/projected/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8-kube-api-access-qnnvr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:21.701970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.701883    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6038ad-db04-4cf3-88d1-874a87a7776d" path="/var/lib/kubelet/pods/1a6038ad-db04-4cf3-88d1-874a87a7776d/volumes"
Oct 30 03:22:21.702531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.702512    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde1591a-2b1e-4c45-b5a5-06f0145e00f6" path="/var/lib/kubelet/pods/fde1591a-2b1e-4c45-b5a5-06f0145e00f6/volumes"
Oct 30 03:22:21.980175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.980079    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" podUID="bb73ac0a-d179-41ed-a82e-f182b7167e60" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.117:50052\" within 1s: context deadline exceeded"
Oct 30 03:22:21.980175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:21.980086    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" podUID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.116:50052\" within 1s: context deadline exceeded"
Oct 30 03:22:22.098261 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.098217    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"
Oct 30 03:22:22.098481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.098266    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8" event={"ID":"3a5a4c67-d88e-4bda-9fa2-ea6393f074b8","Type":"ContainerDied","Data":"f7c8a6a96fbbe49851fe91d1d809817836358a8f2cb5b470247cddf3ec1ce870"}
Oct 30 03:22:22.098481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.098349    4877 scope.go:122] "RemoveContainer" containerID="7cd00a0c8c42c1a73257d0c6890d63df6336227d2af6866eb6ebcca58a121ff0"
Oct 30 03:22:22.101779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.101749    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"
Oct 30 03:22:22.101901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.101773    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp" event={"ID":"bb73ac0a-d179-41ed-a82e-f182b7167e60","Type":"ContainerDied","Data":"0e7aca5bdf9161e4dcd91060fa88b2f9fcf9892da9883789a36076c32a31c9a4"}
Oct 30 03:22:22.115371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.113991    4877 scope.go:122] "RemoveContainer" containerID="deb7292adfaeee5c62ac4b4884039760ce5f622e4dc5bed2a18c379f6e0e32db"
Oct 30 03:22:22.152669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.152635    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"]
Oct 30 03:22:22.164742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.164712    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b132360f-ff47558fd-5rxjp"]
Oct 30 03:22:22.191115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.191070    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"]
Oct 30 03:22:22.196585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.196552    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-qwmj8"]
Oct 30 03:22:22.513594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.513514    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" podUID="ca409c7a-d071-41bb-a21c-a828b8dfe651" containerName="payload-processing" containerID="cri-o://a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e" gracePeriod=25
Oct 30 03:22:22.657672 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:22.657639    4877 logging.go:55] [core] [Channel #2076 SubChannel #2077]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.123:50052", ServerName: "10.133.0.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.123:50052: connect: connection refused"
Oct 30 03:22:22.782163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.782137    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:22:22.877811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.877780    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt\" (UniqueName: \"kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt\") pod \"ca409c7a-d071-41bb-a21c-a828b8dfe651\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") "
Oct 30 03:22:22.878010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.877903    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume\") pod \"ca409c7a-d071-41bb-a21c-a828b8dfe651\" (UID: \"ca409c7a-d071-41bb-a21c-a828b8dfe651\") "
Oct 30 03:22:22.878474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.878329    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume" pod "ca409c7a-d071-41bb-a21c-a828b8dfe651" (UID: "ca409c7a-d071-41bb-a21c-a828b8dfe651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:22.878677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.878646    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409c7a-d071-41bb-a21c-a828b8dfe651-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:22.880641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.880618    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt" pod "ca409c7a-d071-41bb-a21c-a828b8dfe651" (UID: "ca409c7a-d071-41bb-a21c-a828b8dfe651"). InnerVolumeSpecName "kube-api-access-2fxqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:22.980388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:22.980344    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fxqt\" (UniqueName: \"kubernetes.io/projected/ca409c7a-d071-41bb-a21c-a828b8dfe651-kube-api-access-2fxqt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:23.116278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.116178    4877 generic.go:356] "Generic (PLEG): container finished" podID="ca409c7a-d071-41bb-a21c-a828b8dfe651" containerID="a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e" exitCode=0
Oct 30 03:22:23.116278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.116228    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"
Oct 30 03:22:23.116278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.116257    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" event={"ID":"ca409c7a-d071-41bb-a21c-a828b8dfe651","Type":"ContainerDied","Data":"a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e"}
Oct 30 03:22:23.116592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.116320    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" event={"ID":"ca409c7a-d071-41bb-a21c-a828b8dfe651","Type":"ContainerDied","Data":"7af678d6a6f37a9b129e6c406de4d2d9ba232b170a3334957641a79906b848e2"}
Oct 30 03:22:23.116592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.116360    4877 scope.go:122] "RemoveContainer" containerID="a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e"
Oct 30 03:22:23.144226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.144195    4877 scope.go:122] "RemoveContainer" containerID="a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e"
Oct 30 03:22:23.144625 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:23.144593    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e\": container with ID starting with a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e not found: ID does not exist" containerID="a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e"
Oct 30 03:22:23.144703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.144637    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e"} err="failed to get container status \"a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e\": rpc error: code = NotFound desc = could not find container \"a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e\": container with ID starting with a2c113069d894c564f3b2c72ce90631f8b139b494bd15d13d3e7ef584c03519e not found: ID does not exist"
Oct 30 03:22:23.174409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.174369    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"]
Oct 30 03:22:23.187843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.186844    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85"]
Oct 30 03:22:23.425885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.425795    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"]
Oct 30 03:22:23.428461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.426117    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" containerID="cri-o://228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59" gracePeriod=30
Oct 30 03:22:23.657847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.657793    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-58d9b4c9cd-swp85" podUID="ca409c7a-d071-41bb-a21c-a828b8dfe651" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.123:50052\" within 1s: context deadline exceeded"
Oct 30 03:22:23.701953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.701863    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" path="/var/lib/kubelet/pods/3a5a4c67-d88e-4bda-9fa2-ea6393f074b8/volumes"
Oct 30 03:22:23.702351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.702332    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb73ac0a-d179-41ed-a82e-f182b7167e60" path="/var/lib/kubelet/pods/bb73ac0a-d179-41ed-a82e-f182b7167e60/volumes"
Oct 30 03:22:23.702793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:23.702772    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca409c7a-d071-41bb-a21c-a828b8dfe651" path="/var/lib/kubelet/pods/ca409c7a-d071-41bb-a21c-a828b8dfe651/volumes"
Oct 30 03:22:26.801248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:26.801209    4877 patch_prober.go:29] interesting pod/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:22:26.801754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:26.801285    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:22:27.883948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.883906    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"]
Oct 30 03:22:27.884839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.884812    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy"
Oct 30 03:22:27.885018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885002    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy"
Oct 30 03:22:27.885131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885120    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bb73ac0a-d179-41ed-a82e-f182b7167e60" containerName="payload-processing"
Oct 30 03:22:27.885220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885201    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb73ac0a-d179-41ed-a82e-f182b7167e60" containerName="payload-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885241    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1a6038ad-db04-4cf3-88d1-874a87a7776d" containerName="payload-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885249    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6038ad-db04-4cf3-88d1-874a87a7776d" containerName="payload-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885263    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ca409c7a-d071-41bb-a21c-a828b8dfe651" containerName="payload-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885271    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca409c7a-d071-41bb-a21c-a828b8dfe651" containerName="payload-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885282    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885289    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885317    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fde1591a-2b1e-4c45-b5a5-06f0145e00f6" containerName="payload-pre-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885325    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde1591a-2b1e-4c45-b5a5-06f0145e00f6" containerName="payload-pre-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885347    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" containerName="payload-pre-processing"
Oct 30 03:22:27.885360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885355    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" containerName="payload-pre-processing"
Oct 30 03:22:27.885727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885546    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca409c7a-d071-41bb-a21c-a828b8dfe651" containerName="payload-processing"
Oct 30 03:22:27.885727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885562    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy"
Oct 30 03:22:27.885727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885574    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde1591a-2b1e-4c45-b5a5-06f0145e00f6" containerName="payload-pre-processing"
Oct 30 03:22:27.885727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885586    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5a4c67-d88e-4bda-9fa2-ea6393f074b8" containerName="payload-pre-processing"
Oct 30 03:22:27.885727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885601    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb73ac0a-d179-41ed-a82e-f182b7167e60" containerName="payload-processing"
Oct 30 03:22:27.885727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.885614    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6038ad-db04-4cf3-88d1-874a87a7776d" containerName="payload-processing"
Oct 30 03:22:27.893187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.891307    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:27.894900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.894873    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"]
Oct 30 03:22:27.905837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.905810    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6c536\""
Oct 30 03:22:27.917354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:27.917309    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6c536-dockercfg-pm5vp\""
Oct 30 03:22:28.050186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.050156    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtc6f\" (UniqueName: \"kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.050373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.050324    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.151127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.151045    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.151288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.151159    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtc6f\" (UniqueName: \"kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.151817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.151793    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.169960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.169927    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtc6f\" (UniqueName: \"kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.212682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.212651    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:28.520173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.520139    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"]
Oct 30 03:22:28.524829 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:28.524800    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be95dad_ad7f_4b82_93be_f58082f057db.slice/crio-d8ef691fe574b8245f50e58b07e35667ac1ec7b3d65cb7819a2dfea8f69c9053 WatchSource:0}: Error finding container d8ef691fe574b8245f50e58b07e35667ac1ec7b3d65cb7819a2dfea8f69c9053: Status 404 returned error can't find the container with id d8ef691fe574b8245f50e58b07e35667ac1ec7b3d65cb7819a2dfea8f69c9053
Oct 30 03:22:28.541824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.541805    4877 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:22:28.725987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.725961    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:22:28.757651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757620    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.757824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757658    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.757824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757686    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.757824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757719    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.757824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757788    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.758053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757835    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.758053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757869    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:28.758053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.757960    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.758053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758001    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.758053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758021    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:28.758317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758094    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket\") pod \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\" (UID: \"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f\") "
Oct 30 03:22:28.758317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758221    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:28.758536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758503    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:28.758536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758525    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:28.758655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758637    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.758722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758661    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.758722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758677    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.758722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758692    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.758722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.758707    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.759991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.759972    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:22:28.760058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.759998    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:28.760328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.760305    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:28.760458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.760333    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv" pod "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" (UID: "1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f"). InnerVolumeSpecName "kube-api-access-xnwnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:28.859909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.859874    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.859909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.859906    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.859909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.859917    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:28.860148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:28.859925    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnwnv\" (UniqueName: \"kubernetes.io/projected/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f-kube-api-access-xnwnv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:29.168407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.168194    4877 generic.go:356] "Generic (PLEG): container finished" podID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerID="228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59" exitCode=0
Oct 30 03:22:29.168407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.168275    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"
Oct 30 03:22:29.168407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.168277    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" event={"ID":"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f","Type":"ContainerDied","Data":"228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59"}
Oct 30 03:22:29.169106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.168473    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds" event={"ID":"1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f","Type":"ContainerDied","Data":"17603b5150b0ab5b9dbb57b8ad00ec2ba982cd900694263dd72b85ff45a2db3f"}
Oct 30 03:22:29.169106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.168511    4877 scope.go:122] "RemoveContainer" containerID="228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59"
Oct 30 03:22:29.171234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.171173    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx" event={"ID":"0be95dad-ad7f-4b82-93be-f58082f057db","Type":"ContainerStarted","Data":"d8ef691fe574b8245f50e58b07e35667ac1ec7b3d65cb7819a2dfea8f69c9053"}
Oct 30 03:22:29.206848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.206819    4877 scope.go:122] "RemoveContainer" containerID="756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"
Oct 30 03:22:29.226879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.226706    4877 scope.go:122] "RemoveContainer" containerID="228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59"
Oct 30 03:22:29.227390 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:29.227236    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59\": container with ID starting with 228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59 not found: ID does not exist" containerID="228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59"
Oct 30 03:22:29.227390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.227281    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59"} err="failed to get container status \"228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59\": rpc error: code = NotFound desc = could not find container \"228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59\": container with ID starting with 228537d3407c1452fe4133aeab4cbca4699d68e7e93b98349306b60d8e330e59 not found: ID does not exist"
Oct 30 03:22:29.227390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.227300    4877 scope.go:122] "RemoveContainer" containerID="756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"
Oct 30 03:22:29.227792 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:29.227713    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e\": container with ID starting with 756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e not found: ID does not exist" containerID="756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"
Oct 30 03:22:29.227792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.227744    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e"} err="failed to get container status \"756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e\": rpc error: code = NotFound desc = could not find container \"756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e\": container with ID starting with 756db0cd41d98ec1f50eedb3f87b7d372653a41707fed17f62d6c0939657a37e not found: ID does not exist"
Oct 30 03:22:29.239197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.239159    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"]
Oct 30 03:22:29.245521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.245492    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b132360f-openshift-default-75cb596d5c-s5rds"]
Oct 30 03:22:29.701619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:29.701583    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" path="/var/lib/kubelet/pods/1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f/volumes"
Oct 30 03:22:31.191967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:31.191916    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx" event={"ID":"0be95dad-ad7f-4b82-93be-f58082f057db","Type":"ContainerStarted","Data":"0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3"}
Oct 30 03:22:31.192395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:31.192028    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:31.231141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:31.231089    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx" podStartSLOduration=2.638952511 podStartE2EDuration="4.231075439s" totalImagesPullingTime="1.592122928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:31.228693037 +0000 UTC m=+1179.657582456" watchObservedRunningTime="2026-10-30 03:22:31.231075439 +0000 UTC m=+1179.659964858"
Oct 30 03:22:32.204891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.204857    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"]
Oct 30 03:22:32.205537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.205517    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy"
Oct 30 03:22:32.205537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.205537    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy"
Oct 30 03:22:32.205680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.205654    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b58b090-24eb-4272-bd9d-ddb332828f3e" containerName="istio-proxy"
Oct 30 03:22:32.205680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.205662    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy"
Oct 30 03:22:32.209474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.209449    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.223816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.223795    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-499bca-openshift-default-dockercfg-89ggp\""
Oct 30 03:22:32.241832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.241790    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"]
Oct 30 03:22:32.299157 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299118    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299187    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299242    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299314    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299382    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299414    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299536    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299576    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.299729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.299650    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtm7\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400239    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtm7\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400393    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400480    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400556    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400584    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400612    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400644    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400680    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400717    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.400880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400861    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.401009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.400985    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.401160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.401128    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.401297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.401161    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.401365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.401316    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.403187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.403167    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.403501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.403478    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.422613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.422590    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.422701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.422650    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtm7\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7\") pod \"e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") " pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.533683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.525959    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:32.879283 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:32.879252    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c0f2b4_47f1_4138_a91b_9c2aa700d32e.slice/crio-51768b5ec51a4f54144f9593c9ac4105d3ff2c52067300272387842f80b80870 WatchSource:0}: Error finding container 51768b5ec51a4f54144f9593c9ac4105d3ff2c52067300272387842f80b80870: Status 404 returned error can't find the container with id 51768b5ec51a4f54144f9593c9ac4105d3ff2c52067300272387842f80b80870
Oct 30 03:22:32.879399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.879383    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"]
Oct 30 03:22:32.881820 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.881788    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:32.881896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.881851    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:32.881896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:32.881882    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:33.211735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:33.211649    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" event={"ID":"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e","Type":"ContainerStarted","Data":"1ea355187b043e4f78a8542a467dd515d46cf62d00c4e91c94495082388dbf7c"}
Oct 30 03:22:33.211735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:33.211713    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" event={"ID":"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e","Type":"ContainerStarted","Data":"51768b5ec51a4f54144f9593c9ac4105d3ff2c52067300272387842f80b80870"}
Oct 30 03:22:33.279004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:33.278946    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podStartSLOduration=1.2789308670000001 podStartE2EDuration="1.278930867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:33.263706736 +0000 UTC m=+1181.692596155" watchObservedRunningTime="2026-10-30 03:22:33.278930867 +0000 UTC m=+1181.707820285"
Oct 30 03:22:33.526821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:33.526790    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:33.533369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:33.533339    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:34.219125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:34.219095    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:34.220178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:34.220159    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:22:37.827092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.827055    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"]
Oct 30 03:22:37.827698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.827683    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy"
Oct 30 03:22:37.827747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.827700    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy"
Oct 30 03:22:37.827840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.827830    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e16d76a-e3d5-4ca1-8f4d-dae69b24f55f" containerName="istio-proxy"
Oct 30 03:22:37.832463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.832445    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:37.856471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.856440    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"]
Oct 30 03:22:37.862240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.862212    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-499bca\""
Oct 30 03:22:37.862370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.862295    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-499bca-dockercfg-f2snv\""
Oct 30 03:22:37.882087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.882052    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64km2\" (UniqueName: \"kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2\") pod \"payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:37.882405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.882367    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:37.984329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.984282    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64km2\" (UniqueName: \"kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2\") pod \"payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:37.991040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.987741    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:37.991485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.986162    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"]
Oct 30 03:22:37.992322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.992289    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:37.999937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.996563    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"]
Oct 30 03:22:37.999937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:37.996715    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.006587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.006566    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64km2\" (UniqueName: \"kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2\") pod \"payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:38.092577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.092479    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.092716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.092674    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bh5\" (UniqueName: \"kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5\") pod \"payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.147199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.147162    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:38.193623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.193586    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bh5\" (UniqueName: \"kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5\") pod \"payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.193822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.193752    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.194668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.194642    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.213862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.213830    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bh5\" (UniqueName: \"kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5\") pod \"payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.357360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.357269    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:38.459308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.459276    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"]
Oct 30 03:22:38.459903 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:38.459875    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9520f7b4_aff0_4116_87a8_edd8647d93ac.slice/crio-5772e86c3fa3e8272c863ddcf128a0cc4a7e6caf2f03dad19f34e9e17270dc80 WatchSource:0}: Error finding container 5772e86c3fa3e8272c863ddcf128a0cc4a7e6caf2f03dad19f34e9e17270dc80: Status 404 returned error can't find the container with id 5772e86c3fa3e8272c863ddcf128a0cc4a7e6caf2f03dad19f34e9e17270dc80
Oct 30 03:22:38.677248 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:38.677213    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e9d020_043c_413f_a6c9_645a721c2d0f.slice/crio-c8cd068d19941c92b490b6cb9bd6f599d3d57eb52a504aaac5cd0a251e1d1f75 WatchSource:0}: Error finding container c8cd068d19941c92b490b6cb9bd6f599d3d57eb52a504aaac5cd0a251e1d1f75: Status 404 returned error can't find the container with id c8cd068d19941c92b490b6cb9bd6f599d3d57eb52a504aaac5cd0a251e1d1f75
Oct 30 03:22:38.678018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.677975    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"]
Oct 30 03:22:38.975881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:38.975794    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"]
Oct 30 03:22:39.262750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.262660    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5" event={"ID":"79e9d020-043c-413f-a6c9-645a721c2d0f","Type":"ContainerStarted","Data":"685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31"}
Oct 30 03:22:39.262750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.262731    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5" event={"ID":"79e9d020-043c-413f-a6c9-645a721c2d0f","Type":"ContainerStarted","Data":"c8cd068d19941c92b490b6cb9bd6f599d3d57eb52a504aaac5cd0a251e1d1f75"}
Oct 30 03:22:39.262979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.262764    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:39.264270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.264243    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj" event={"ID":"9520f7b4-aff0-4116-87a8-edd8647d93ac","Type":"ContainerStarted","Data":"c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b"}
Oct 30 03:22:39.264379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.264292    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj" event={"ID":"9520f7b4-aff0-4116-87a8-edd8647d93ac","Type":"ContainerStarted","Data":"5772e86c3fa3e8272c863ddcf128a0cc4a7e6caf2f03dad19f34e9e17270dc80"}
Oct 30 03:22:39.264441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.264388    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:39.306233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.306164    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5" podStartSLOduration=2.306144623 podStartE2EDuration="2.306144623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:39.30505872 +0000 UTC m=+1187.733948163" watchObservedRunningTime="2026-10-30 03:22:39.306144623 +0000 UTC m=+1187.735034042"
Oct 30 03:22:39.345907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:39.345840    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj" podStartSLOduration=2.34582229 podStartE2EDuration="2.34582229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:39.344784916 +0000 UTC m=+1187.773674335" watchObservedRunningTime="2026-10-30 03:22:39.34582229 +0000 UTC m=+1187.774711708"
Oct 30 03:22:42.204955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:42.204921    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:22:43.983450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:43.980506    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp" podUID="947cd9d9-bafe-4ba8-8ca1-218163942042" containerName="payload-pre-processing" containerID="cri-o://5aba779c3a5ca1b707ae52c814996f4c0e827b4c2af533ff4535378cc9684c27" gracePeriod=25
Oct 30 03:22:44.454698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.454579    4877 generic.go:356] "Generic (PLEG): container finished" podID="947cd9d9-bafe-4ba8-8ca1-218163942042" containerID="5aba779c3a5ca1b707ae52c814996f4c0e827b4c2af533ff4535378cc9684c27" exitCode=0
Oct 30 03:22:44.454698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.454637    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp" event={"ID":"947cd9d9-bafe-4ba8-8ca1-218163942042","Type":"ContainerDied","Data":"5aba779c3a5ca1b707ae52c814996f4c0e827b4c2af533ff4535378cc9684c27"}
Oct 30 03:22:44.488581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.488542    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:22:44.488856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.488813    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:44.917483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.916889    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6vq44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:22:44.917483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.916981    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:44.918360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.918269    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 03:22:44.982451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:44.920000    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:45.056829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.017638    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:45.131116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.129650    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:22:45.456538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.455765    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:45.602448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.598718    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume\") pod \"947cd9d9-bafe-4ba8-8ca1-218163942042\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") "
Oct 30 03:22:45.602448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.598822    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j\" (UniqueName: \"kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j\") pod \"947cd9d9-bafe-4ba8-8ca1-218163942042\" (UID: \"947cd9d9-bafe-4ba8-8ca1-218163942042\") "
Oct 30 03:22:45.684453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.667549    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume" pod "947cd9d9-bafe-4ba8-8ca1-218163942042" (UID: "947cd9d9-bafe-4ba8-8ca1-218163942042"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:45.688743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.688702    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp" event={"ID":"947cd9d9-bafe-4ba8-8ca1-218163942042","Type":"ContainerDied","Data":"f003c1414fb1b94e98f2907ec85da0c6c03ec2118f89fa17b03dd3f2b834c8b0"}
Oct 30 03:22:45.689080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.689057    4877 scope.go:122] "RemoveContainer" containerID="5aba779c3a5ca1b707ae52c814996f4c0e827b4c2af533ff4535378cc9684c27"
Oct 30 03:22:45.692765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.692739    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"
Oct 30 03:22:45.722855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.722754    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947cd9d9-bafe-4ba8-8ca1-218163942042-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:45.737674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.735565    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:45.939635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.939590    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 03:22:45.950098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.950075    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:22:45.953604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.953571    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"]
Oct 30 03:22:45.959611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.959549    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j" pod "947cd9d9-bafe-4ba8-8ca1-218163942042" (UID: "947cd9d9-bafe-4ba8-8ca1-218163942042"). InnerVolumeSpecName "kube-api-access-s8t2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:45.982572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.982488    4877 cpu_manager.go:414] "containerMap: removing container" podUID="947cd9d9-bafe-4ba8-8ca1-218163942042" containerName="payload-pre-processing"
Oct 30 03:22:45.982771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.982750    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947cd9d9-bafe-4ba8-8ca1-218163942042" containerName="payload-pre-processing"
Oct 30 03:22:45.983154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.983132    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="947cd9d9-bafe-4ba8-8ca1-218163942042" containerName="payload-pre-processing"
Oct 30 03:22:45.988711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.988687    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"]
Oct 30 03:22:45.988978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:45.988962    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.022061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.009691    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:22:46.022061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.009871    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:22:46.055599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.049843    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8t2j\" (UniqueName: \"kubernetes.io/projected/947cd9d9-bafe-4ba8-8ca1-218163942042-kube-api-access-s8t2j\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:46.077794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.064656    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:46.077794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.065010    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 03:22:46.077794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.065636    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:22:46.077794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.065694    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:22:46.141735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.140197    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:46.141735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.140539    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:46.141735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.140905    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:22:46.166487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.157827    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.195870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.195833    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n57\" (UniqueName: \"kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.196791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.196765    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.315457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.299216    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n57\" (UniqueName: \"kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.315457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.299303    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.351544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.332493    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.351544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.333733    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.397296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.394174    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n57\" (UniqueName: \"kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.520386 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.520354    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce3839f-qnsqn\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.571499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.567551    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:46.580762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.575531    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:22:46.684781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.683234    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:46.759822 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:22:46.758616    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:22:46.761009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.760980    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:46.854214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.846918    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:22:46.944176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.944144    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:22:46.961442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.953592    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:22:46.961442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.953639    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:22:46.961442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.952518    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:22:46.961442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.958586    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"]
Oct 30 03:22:46.966722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.966381    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-589b6d6f9f-8lmdp"]
Oct 30 03:22:46.976271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:46.976097    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:22:47.130003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:47.127559    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"]
Oct 30 03:22:47.138243 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:47.138214    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc14c9e_6636_4e47_8b22_5fdc45d142d8.slice/crio-7af09c7bf2e8c476b34f5ef3ea976a171c13c65a27ec66531d125a5d4394b420 WatchSource:0}: Error finding container 7af09c7bf2e8c476b34f5ef3ea976a171c13c65a27ec66531d125a5d4394b420: Status 404 returned error can't find the container with id 7af09c7bf2e8c476b34f5ef3ea976a171c13c65a27ec66531d125a5d4394b420
Oct 30 03:22:47.719179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:47.718614    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947cd9d9-bafe-4ba8-8ca1-218163942042" path="/var/lib/kubelet/pods/947cd9d9-bafe-4ba8-8ca1-218163942042/volumes"
Oct 30 03:22:47.975065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:47.974964    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn" event={"ID":"0dc14c9e-6636-4e47-8b22-5fdc45d142d8","Type":"ContainerStarted","Data":"5233cf5cdcc600b246f940c9b35f4351321bc27a5b999d4c2e78236ad7c3a761"}
Oct 30 03:22:47.975065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:47.975032    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn" event={"ID":"0dc14c9e-6636-4e47-8b22-5fdc45d142d8","Type":"ContainerStarted","Data":"7af09c7bf2e8c476b34f5ef3ea976a171c13c65a27ec66531d125a5d4394b420"}
Oct 30 03:22:48.025045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:48.024985    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn" podStartSLOduration=3.02497193 podStartE2EDuration="3.02497193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:48.023115696 +0000 UTC m=+1196.452005139" watchObservedRunningTime="2026-10-30 03:22:48.02497193 +0000 UTC m=+1196.453861349"
Oct 30 03:22:49.006821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:49.006784    4877 generic.go:356] "Generic (PLEG): container finished" podID="0dc14c9e-6636-4e47-8b22-5fdc45d142d8" containerID="5233cf5cdcc600b246f940c9b35f4351321bc27a5b999d4c2e78236ad7c3a761" exitCode=0
Oct 30 03:22:49.007282 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:49.006863    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn" event={"ID":"0dc14c9e-6636-4e47-8b22-5fdc45d142d8","Type":"ContainerDied","Data":"5233cf5cdcc600b246f940c9b35f4351321bc27a5b999d4c2e78236ad7c3a761"}
Oct 30 03:22:50.179019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.178984    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:50.185528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.185505    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57\" (UniqueName: \"kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57\") pod \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") "
Oct 30 03:22:50.185669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.185654    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca\") pod \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") "
Oct 30 03:22:50.185712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.185694    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca\") pod \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\" (UID: \"0dc14c9e-6636-4e47-8b22-5fdc45d142d8\") "
Oct 30 03:22:50.186153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.186117    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca" pod "0dc14c9e-6636-4e47-8b22-5fdc45d142d8" (UID: "0dc14c9e-6636-4e47-8b22-5fdc45d142d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:22:50.186323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.186299    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:50.187738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.187711    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57" pod "0dc14c9e-6636-4e47-8b22-5fdc45d142d8" (UID: "0dc14c9e-6636-4e47-8b22-5fdc45d142d8"). InnerVolumeSpecName "kube-api-access-67n57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:50.188008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.187986    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca" pod "0dc14c9e-6636-4e47-8b22-5fdc45d142d8" (UID: "0dc14c9e-6636-4e47-8b22-5fdc45d142d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:50.274448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.274393    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:22:50.274633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.274495    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:22:50.288227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.288166    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n57\" (UniqueName: \"kubernetes.io/projected/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-kube-api-access-67n57\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:50.288227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.288201    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc14c9e-6636-4e47-8b22-5fdc45d142d8-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:22:50.990986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.990947    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"]
Oct 30 03:22:50.991724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.991692    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0dc14c9e-6636-4e47-8b22-5fdc45d142d8" containerName="revoke-keys"
Oct 30 03:22:50.991724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.991716    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc14c9e-6636-4e47-8b22-5fdc45d142d8" containerName="revoke-keys"
Oct 30 03:22:50.991977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.991962    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc14c9e-6636-4e47-8b22-5fdc45d142d8" containerName="revoke-keys"
Oct 30 03:22:50.995896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:50.995875    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.016511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.016485    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"]
Oct 30 03:22:51.030542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.030456    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn" event={"ID":"0dc14c9e-6636-4e47-8b22-5fdc45d142d8","Type":"ContainerDied","Data":"7af09c7bf2e8c476b34f5ef3ea976a171c13c65a27ec66531d125a5d4394b420"}
Oct 30 03:22:51.030542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.030515    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af09c7bf2e8c476b34f5ef3ea976a171c13c65a27ec66531d125a5d4394b420"
Oct 30 03:22:51.030542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.030540    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"
Oct 30 03:22:51.116207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.099814    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a1e5d5-dockercfg-p27w9\""
Oct 30 03:22:51.116207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.099912    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1e5d5\""
Oct 30 03:22:51.202796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.202763    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9sj\" (UniqueName: \"kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.203275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.202893    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.304315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.304269    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9sj\" (UniqueName: \"kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.304539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.304461    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.305191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.305167    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.323702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.323671    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9sj\" (UniqueName: \"kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.610444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.610344    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:51.920702 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:51.920669    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c3bf4f_d669_4dbb_b217_ac508c47db91.slice/crio-eeb82cee539c8b135131250dfefe5343804d3f1ecaf50a2074df75afa3478110 WatchSource:0}: Error finding container eeb82cee539c8b135131250dfefe5343804d3f1ecaf50a2074df75afa3478110: Status 404 returned error can't find the container with id eeb82cee539c8b135131250dfefe5343804d3f1ecaf50a2074df75afa3478110
Oct 30 03:22:51.922270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:51.922231    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"]
Oct 30 03:22:52.061810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.061778    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f" event={"ID":"24c3bf4f-d669-4dbb-b217-ac508c47db91","Type":"ContainerStarted","Data":"eeb82cee539c8b135131250dfefe5343804d3f1ecaf50a2074df75afa3478110"}
Oct 30 03:22:52.062838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.062815    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:22:52.089704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.089672    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:22:52.282636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.282601    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"]
Oct 30 03:22:52.288881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.288855    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.305169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.305136    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"]
Oct 30 03:22:52.312971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.312938    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04fdf3\""
Oct 30 03:22:52.313134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.313021    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04fdf3-dockercfg-69whl\""
Oct 30 03:22:52.318909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.318877    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmkm\" (UniqueName: \"kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.319043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.319005    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.420368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.420324    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmkm\" (UniqueName: \"kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.420571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.420398    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.421327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.421304    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.439673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.439647    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmkm\" (UniqueName: \"kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm\") pod \"payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.447631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.447604    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"]
Oct 30 03:22:52.452323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.452288    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.476332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.476300    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"]
Oct 30 03:22:52.521535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.521499    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.521728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.521689    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lrx\" (UniqueName: \"kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx\") pod \"payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.607861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.607771    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:52.623015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.622975    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.623201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.623109    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lrx\" (UniqueName: \"kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx\") pod \"payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.623829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.623806    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.643817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.643791    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lrx\" (UniqueName: \"kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx\") pod \"payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.775546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.775502    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:52.964355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:52.964318    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"]
Oct 30 03:22:52.964500 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:22:52.964317    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7cb16e_37e3_4463_877d_efae398806f0.slice/crio-fff85a356658c016ab001b9dab6b0975af2ce670bd089142aa5a4225de3919a6 WatchSource:0}: Error finding container fff85a356658c016ab001b9dab6b0975af2ce670bd089142aa5a4225de3919a6: Status 404 returned error can't find the container with id fff85a356658c016ab001b9dab6b0975af2ce670bd089142aa5a4225de3919a6
Oct 30 03:22:53.072489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.072455    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv" event={"ID":"3c7cb16e-37e3-4463-877d-efae398806f0","Type":"ContainerStarted","Data":"7d2f10cb3613a6cc380a0fc16c27cb39d4e3bbd091058e3edf229b1a017686fe"}
Oct 30 03:22:53.080385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.072526    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv" event={"ID":"3c7cb16e-37e3-4463-877d-efae398806f0","Type":"ContainerStarted","Data":"fff85a356658c016ab001b9dab6b0975af2ce670bd089142aa5a4225de3919a6"}
Oct 30 03:22:53.080385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.072591    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:22:53.080385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.074315    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f" event={"ID":"24c3bf4f-d669-4dbb-b217-ac508c47db91","Type":"ContainerStarted","Data":"b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394"}
Oct 30 03:22:53.080385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.074471    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:22:53.130607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.129928    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"]
Oct 30 03:22:53.133777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.133718    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv" podStartSLOduration=1.133695586 podStartE2EDuration="1.133695586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:53.129682804 +0000 UTC m=+1201.558572207" watchObservedRunningTime="2026-10-30 03:22:53.133695586 +0000 UTC m=+1201.562585009"
Oct 30 03:22:53.186105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:53.186041    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f" podStartSLOduration=3.186022795 podStartE2EDuration="3.186022795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:53.181294398 +0000 UTC m=+1201.610183816" watchObservedRunningTime="2026-10-30 03:22:53.186022795 +0000 UTC m=+1201.614912212"
Oct 30 03:22:54.104473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.104387    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq" event={"ID":"502500f2-9ce1-47e9-88f7-1649164bd142","Type":"ContainerStarted","Data":"9a2c72bd6ef946701ceb18cb03c4c75b0ffbc71540fe52e2cbbc7adb4ead60d3"}
Oct 30 03:22:54.104989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.104487    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq" event={"ID":"502500f2-9ce1-47e9-88f7-1649164bd142","Type":"ContainerStarted","Data":"41847bdc805dca048fd9a6a5beff008e5cd60d50027e69d1a482607950a795c2"}
Oct 30 03:22:54.104989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.104540    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:22:54.109104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.109077    4877 generic.go:356] "Generic (PLEG): container finished" podID="075ae435-693b-406e-ad16-3d253430068b" containerID="b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1" exitCode=0
Oct 30 03:22:54.109380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.109148    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" event={"ID":"075ae435-693b-406e-ad16-3d253430068b","Type":"ContainerDied","Data":"b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"}
Oct 30 03:22:54.109928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.109900    4877 scope.go:122] "RemoveContainer" containerID="b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"
Oct 30 03:22:54.112966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.111704    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:54.112966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.111793    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:54.112966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.111837    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:22:54.151039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:54.150994    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq" podStartSLOduration=2.150981488 podStartE2EDuration="2.150981488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:54.146823868 +0000 UTC m=+1202.575713296" watchObservedRunningTime="2026-10-30 03:22:54.150981488 +0000 UTC m=+1202.579870906"
Oct 30 03:22:55.123501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:55.123453    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" event={"ID":"075ae435-693b-406e-ad16-3d253430068b","Type":"ContainerStarted","Data":"59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89"}
Oct 30 03:22:55.649853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:55.649810    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:22:55.650069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:55.650051    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 03:22:55.650131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:55.650096    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 03:22:56.649991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:56.649958    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 03:22:56.650393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:56.650010    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 03:22:57.649644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:57.649606    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 03:22:57.649818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:57.649660    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 03:22:58.649356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:58.649318    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 03:22:58.649794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:58.649379    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 03:22:59.649685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:59.649650    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 03:22:59.650105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:22:59.649703    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 03:23:00.687446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:00.686481    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:23:00.687904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:00.687518    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:23:00.689075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:00.689040    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:23:02.099700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.099660    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"]
Oct 30 03:23:02.107215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.107190    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.123557 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.123521    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"]
Oct 30 03:23:02.173975 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.173940    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6c106c\""
Oct 30 03:23:02.174297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.174175    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6c106c-dockercfg-gf2hs\""
Oct 30 03:23:02.228448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.226669    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.228448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.226749    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4twqz\" (UniqueName: \"kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.327763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.327724    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.327949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.327852    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4twqz\" (UniqueName: \"kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.328413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.328391    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.347078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.347053    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4twqz\" (UniqueName: \"kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.426586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.426497    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:02.729478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:02.729443    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"]
Oct 30 03:23:02.730067 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:02.730033    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a24968_074a_401c_906c_788c25f64566.slice/crio-f68e7269d1515079936e28d8d98b1191380f2095cbb8f31996bd86dd3acdebd8 WatchSource:0}: Error finding container f68e7269d1515079936e28d8d98b1191380f2095cbb8f31996bd86dd3acdebd8: Status 404 returned error can't find the container with id f68e7269d1515079936e28d8d98b1191380f2095cbb8f31996bd86dd3acdebd8
Oct 30 03:23:03.247403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:03.247364    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7" event={"ID":"76a24968-074a-401c-906c-788c25f64566","Type":"ContainerStarted","Data":"6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804"}
Oct 30 03:23:03.247930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:03.247419    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7" event={"ID":"76a24968-074a-401c-906c-788c25f64566","Type":"ContainerStarted","Data":"f68e7269d1515079936e28d8d98b1191380f2095cbb8f31996bd86dd3acdebd8"}
Oct 30 03:23:03.247930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:03.247496    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:03.290693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:03.289900    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7" podStartSLOduration=1.289880925 podStartE2EDuration="1.289880925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:03.289561377 +0000 UTC m=+1211.718450822" watchObservedRunningTime="2026-10-30 03:23:03.289880925 +0000 UTC m=+1211.718770344"
Oct 30 03:23:04.113206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:04.113123    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:23:04.113996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:04.113973    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:23:05.126627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:05.126591    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:23:12.049087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.049042    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"]
Oct 30 03:23:12.053349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.053323    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.073158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.073132    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"]
Oct 30 03:23:12.079253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.079210    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0f003\""
Oct 30 03:23:12.079384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.079221    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0f003-dockercfg-nc729\""
Oct 30 03:23:12.154213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.154167    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45q2q\" (UniqueName: \"kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q\") pod \"payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.154473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.154447    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.259597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.259557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.260904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.260877    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45q2q\" (UniqueName: \"kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q\") pod \"payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.265580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.260684    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.281182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.281155    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45q2q\" (UniqueName: \"kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q\") pod \"payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.368725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.368637    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:12.685746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:12.685718    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"]
Oct 30 03:23:12.685990 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:12.685957    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031b3817_c93f_4318_b662_2cea5fff7c71.slice/crio-1c241bb8c6d0f360816f651b98fa123965b34296050cf6f419f7e9515179be17 WatchSource:0}: Error finding container 1c241bb8c6d0f360816f651b98fa123965b34296050cf6f419f7e9515179be17: Status 404 returned error can't find the container with id 1c241bb8c6d0f360816f651b98fa123965b34296050cf6f419f7e9515179be17
Oct 30 03:23:13.343117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:13.343067    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt" event={"ID":"031b3817-c93f-4318-b662-2cea5fff7c71","Type":"ContainerStarted","Data":"204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb"}
Oct 30 03:23:13.343540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:13.343179    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt" event={"ID":"031b3817-c93f-4318-b662-2cea5fff7c71","Type":"ContainerStarted","Data":"1c241bb8c6d0f360816f651b98fa123965b34296050cf6f419f7e9515179be17"}
Oct 30 03:23:13.343540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:13.343216    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:13.387418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:13.387361    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt" podStartSLOduration=1.387345358 podStartE2EDuration="1.387345358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:13.386522864 +0000 UTC m=+1221.815412294" watchObservedRunningTime="2026-10-30 03:23:13.387345358 +0000 UTC m=+1221.816234775"
Oct 30 03:23:14.258641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:14.258612    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:16.236715 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:23:16.236672    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:23:16.311498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:16.311453    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"]
Oct 30 03:23:16.330652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:16.330615    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"]
Oct 30 03:23:21.314495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.313774    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5" podUID="79e9d020-043c-413f-a6c9-645a721c2d0f" containerName="payload-processing" containerID="cri-o://685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31" gracePeriod=25
Oct 30 03:23:21.331186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.331131    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj" podUID="9520f7b4-aff0-4116-87a8-edd8647d93ac" containerName="payload-pre-processing" containerID="cri-o://c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b" gracePeriod=25
Oct 30 03:23:21.649178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.649142    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:23:21.677556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.677523    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:23:21.726981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.726942    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2\" (UniqueName: \"kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2\") pod \"9520f7b4-aff0-4116-87a8-edd8647d93ac\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") "
Oct 30 03:23:21.727240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.727066    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume\") pod \"79e9d020-043c-413f-a6c9-645a721c2d0f\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") "
Oct 30 03:23:21.727240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.727119    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5\" (UniqueName: \"kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5\") pod \"79e9d020-043c-413f-a6c9-645a721c2d0f\" (UID: \"79e9d020-043c-413f-a6c9-645a721c2d0f\") "
Oct 30 03:23:21.727240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.727226    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume\") pod \"9520f7b4-aff0-4116-87a8-edd8647d93ac\" (UID: \"9520f7b4-aff0-4116-87a8-edd8647d93ac\") "
Oct 30 03:23:21.728118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.728086    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume" pod "9520f7b4-aff0-4116-87a8-edd8647d93ac" (UID: "9520f7b4-aff0-4116-87a8-edd8647d93ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:21.749319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.749285    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume" pod "79e9d020-043c-413f-a6c9-645a721c2d0f" (UID: "79e9d020-043c-413f-a6c9-645a721c2d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:21.757761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.757732    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5" pod "79e9d020-043c-413f-a6c9-645a721c2d0f" (UID: "79e9d020-043c-413f-a6c9-645a721c2d0f"). InnerVolumeSpecName "kube-api-access-s9bh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:21.776639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.776603    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2" pod "9520f7b4-aff0-4116-87a8-edd8647d93ac" (UID: "9520f7b4-aff0-4116-87a8-edd8647d93ac"). InnerVolumeSpecName "kube-api-access-64km2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:21.829855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.828932    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9520f7b4-aff0-4116-87a8-edd8647d93ac-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:21.829855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.829339    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64km2\" (UniqueName: \"kubernetes.io/projected/9520f7b4-aff0-4116-87a8-edd8647d93ac-kube-api-access-64km2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:21.829855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.829521    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9d020-043c-413f-a6c9-645a721c2d0f-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:21.829855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:21.829546    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9bh5\" (UniqueName: \"kubernetes.io/projected/79e9d020-043c-413f-a6c9-645a721c2d0f-kube-api-access-s9bh5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:22.443531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.443490    4877 generic.go:356] "Generic (PLEG): container finished" podID="79e9d020-043c-413f-a6c9-645a721c2d0f" containerID="685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31" exitCode=0
Oct 30 03:23:22.443976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.443545    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5" event={"ID":"79e9d020-043c-413f-a6c9-645a721c2d0f","Type":"ContainerDied","Data":"685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31"}
Oct 30 03:23:22.443976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.443564    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"
Oct 30 03:23:22.443976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.443652    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5" event={"ID":"79e9d020-043c-413f-a6c9-645a721c2d0f","Type":"ContainerDied","Data":"c8cd068d19941c92b490b6cb9bd6f599d3d57eb52a504aaac5cd0a251e1d1f75"}
Oct 30 03:23:22.443976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.443686    4877 scope.go:122] "RemoveContainer" containerID="685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31"
Oct 30 03:23:22.445486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.445449    4877 generic.go:356] "Generic (PLEG): container finished" podID="9520f7b4-aff0-4116-87a8-edd8647d93ac" containerID="c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b" exitCode=0
Oct 30 03:23:22.445630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.445512    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj" event={"ID":"9520f7b4-aff0-4116-87a8-edd8647d93ac","Type":"ContainerDied","Data":"c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b"}
Oct 30 03:23:22.445630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.445572    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj" event={"ID":"9520f7b4-aff0-4116-87a8-edd8647d93ac","Type":"ContainerDied","Data":"5772e86c3fa3e8272c863ddcf128a0cc4a7e6caf2f03dad19f34e9e17270dc80"}
Oct 30 03:23:22.445630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.445521    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"
Oct 30 03:23:22.456944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.456919    4877 scope.go:122] "RemoveContainer" containerID="685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31"
Oct 30 03:23:22.457273 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:23:22.457246    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31\": container with ID starting with 685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31 not found: ID does not exist" containerID="685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31"
Oct 30 03:23:22.457355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.457285    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31"} err="failed to get container status \"685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31\": rpc error: code = NotFound desc = could not find container \"685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31\": container with ID starting with 685ab6764d0fc03aa88019fd2e71ab6f992d2b7567b27bd3932e16e5d877ba31 not found: ID does not exist"
Oct 30 03:23:22.457355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.457309    4877 scope.go:122] "RemoveContainer" containerID="c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b"
Oct 30 03:23:22.466954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.466940    4877 scope.go:122] "RemoveContainer" containerID="c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b"
Oct 30 03:23:22.467193 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:23:22.467176    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b\": container with ID starting with c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b not found: ID does not exist" containerID="c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b"
Oct 30 03:23:22.467254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.467200    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b"} err="failed to get container status \"c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b\": rpc error: code = NotFound desc = could not find container \"c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b\": container with ID starting with c9148efa2eb73fd112989b0c4df85cbb2a2e74951c2f25f778e923cfdd87dd5b not found: ID does not exist"
Oct 30 03:23:22.505469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.500635    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"]
Oct 30 03:23:22.511163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.511118    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-5498f77768-4fhh5"]
Oct 30 03:23:22.542995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.542958    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"]
Oct 30 03:23:22.546525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:22.546496    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-7fd5ccbdd7-qzjmj"]
Oct 30 03:23:23.700510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:23.700477    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e9d020-043c-413f-a6c9-645a721c2d0f" path="/var/lib/kubelet/pods/79e9d020-043c-413f-a6c9-645a721c2d0f/volumes"
Oct 30 03:23:23.701011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:23.700851    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9520f7b4-aff0-4116-87a8-edd8647d93ac" path="/var/lib/kubelet/pods/9520f7b4-aff0-4116-87a8-edd8647d93ac/volumes"
Oct 30 03:23:24.353183 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:24.353155    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:23:25.473467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:25.473405    4877 generic.go:356] "Generic (PLEG): container finished" podID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerID="1ea355187b043e4f78a8542a467dd515d46cf62d00c4e91c94495082388dbf7c" exitCode=0
Oct 30 03:23:25.474030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:25.473485    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" event={"ID":"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e","Type":"ContainerDied","Data":"1ea355187b043e4f78a8542a467dd515d46cf62d00c4e91c94495082388dbf7c"}
Oct 30 03:23:25.474030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:25.473857    4877 scope.go:122] "RemoveContainer" containerID="1ea355187b043e4f78a8542a467dd515d46cf62d00c4e91c94495082388dbf7c"
Oct 30 03:23:25.476026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:25.475976    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:23:25.476171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:25.476145    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:23:25.476274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:25.476249    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:23:26.511444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:26.511300    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" event={"ID":"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e","Type":"ContainerStarted","Data":"7612037a50cbb104dc0294a08c9eb1ffc39319686710b4f6f2ba642b416f94ff"}
Oct 30 03:23:26.527159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:26.527129    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:23:26.527350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:26.527326    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 03:23:26.527406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:26.527382    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 03:23:27.526992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:27.526953    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 03:23:27.527386 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:27.527026    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 03:23:28.526854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:28.526815    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 03:23:28.527067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:28.526876    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 03:23:29.527190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:29.527128    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 03:23:29.527617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:29.527226    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 03:23:30.527577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:30.527534    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 03:23:30.528087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:30.527616    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 03:23:31.527105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:31.527057    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 03:23:31.527286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:31.527131    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 03:23:32.531509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:32.531479    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:23:32.531998 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:32.531931    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:23:32.532720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:32.532699    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:23:34.952697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:34.952663    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"]
Oct 30 03:23:37.725496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.725449    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"]
Oct 30 03:23:37.726989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.726958    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9520f7b4-aff0-4116-87a8-edd8647d93ac" containerName="payload-pre-processing"
Oct 30 03:23:37.727999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.727976    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9520f7b4-aff0-4116-87a8-edd8647d93ac" containerName="payload-pre-processing"
Oct 30 03:23:37.728239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.728225    4877 cpu_manager.go:414] "containerMap: removing container" podUID="79e9d020-043c-413f-a6c9-645a721c2d0f" containerName="payload-processing"
Oct 30 03:23:37.728786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.728767    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e9d020-043c-413f-a6c9-645a721c2d0f" containerName="payload-processing"
Oct 30 03:23:37.729055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.729039    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9520f7b4-aff0-4116-87a8-edd8647d93ac" containerName="payload-pre-processing"
Oct 30 03:23:37.729178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.729166    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e9d020-043c-413f-a6c9-645a721c2d0f" containerName="payload-processing"
Oct 30 03:23:37.738854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.738783    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.753498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.753459    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"]
Oct 30 03:23:37.758883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.758858    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-499bca-dockercfg-2zjlb\""
Oct 30 03:23:37.759039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.758877    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-499bca\""
Oct 30 03:23:37.769104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.769059    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.769270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.769140    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfcs\" (UniqueName: \"kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs\") pod \"payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.870084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.870039    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.870307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.870106    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfcs\" (UniqueName: \"kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs\") pod \"payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.871064 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.871030    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.877377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.877342    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"]
Oct 30 03:23:37.883226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.883198    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:37.890942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.890912    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfcs\" (UniqueName: \"kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs\") pod \"payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:37.899997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.899968    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"]
Oct 30 03:23:37.970767 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.970732    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:37.970948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:37.970806    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtvt\" (UniqueName: \"kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt\") pod \"payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.059680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.059643    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:38.072373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.072340    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.072507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.072380    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtvt\" (UniqueName: \"kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt\") pod \"payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.073216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.073137    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.092009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.091979    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtvt\" (UniqueName: \"kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt\") pod \"payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.233993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.233960    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.369369 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:38.369317    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc2c451_934c_4ca0_b73a_9717f242250a.slice/crio-8aa2da98eb13e0ff2012a4babed01e5712765129ebb2b729ae1017ccdf5e781b WatchSource:0}: Error finding container 8aa2da98eb13e0ff2012a4babed01e5712765129ebb2b729ae1017ccdf5e781b: Status 404 returned error can't find the container with id 8aa2da98eb13e0ff2012a4babed01e5712765129ebb2b729ae1017ccdf5e781b
Oct 30 03:23:38.373572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.372991    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"]
Oct 30 03:23:38.543650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.543621    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"]
Oct 30 03:23:38.543897 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:38.543868    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dcfaa7_1a91_4f6b_8d9c_0705b1b2dcbc.slice/crio-ef2a561ed123e99905a93df0989c250ee7d1d9ba77b8648a9ed722f5cf5a957c WatchSource:0}: Error finding container ef2a561ed123e99905a93df0989c250ee7d1d9ba77b8648a9ed722f5cf5a957c: Status 404 returned error can't find the container with id ef2a561ed123e99905a93df0989c250ee7d1d9ba77b8648a9ed722f5cf5a957c
Oct 30 03:23:38.660610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.660568    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779" event={"ID":"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc","Type":"ContainerStarted","Data":"a83ab68734843a90613a5bf6626c8884cbe0048d2796fdb24ca20544235ca63f"}
Oct 30 03:23:38.661073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.660809    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:38.661073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.660855    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779" event={"ID":"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc","Type":"ContainerStarted","Data":"ef2a561ed123e99905a93df0989c250ee7d1d9ba77b8648a9ed722f5cf5a957c"}
Oct 30 03:23:38.662236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.662208    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs" event={"ID":"9fc2c451-934c-4ca0-b73a-9717f242250a","Type":"ContainerStarted","Data":"da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f"}
Oct 30 03:23:38.662363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.662254    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs" event={"ID":"9fc2c451-934c-4ca0-b73a-9717f242250a","Type":"ContainerStarted","Data":"8aa2da98eb13e0ff2012a4babed01e5712765129ebb2b729ae1017ccdf5e781b"}
Oct 30 03:23:38.662363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.662298    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:38.700184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.700135    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779" podStartSLOduration=1.700121732 podStartE2EDuration="1.700121732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:38.69919638 +0000 UTC m=+1247.128085805" watchObservedRunningTime="2026-10-30 03:23:38.700121732 +0000 UTC m=+1247.129011198"
Oct 30 03:23:38.741677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:38.741627    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs" podStartSLOduration=1.741613007 podStartE2EDuration="1.741613007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:38.738187531 +0000 UTC m=+1247.167076961" watchObservedRunningTime="2026-10-30 03:23:38.741613007 +0000 UTC m=+1247.170502456"
Oct 30 03:23:39.953881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:39.953842    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f" podUID="24c3bf4f-d669-4dbb-b217-ac508c47db91" containerName="payload-pre-processing" containerID="cri-o://b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394" gracePeriod=25
Oct 30 03:23:40.246898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.246860    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:23:40.303116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.303082    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume\") pod \"24c3bf4f-d669-4dbb-b217-ac508c47db91\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") "
Oct 30 03:23:40.303323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.303210    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj\" (UniqueName: \"kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj\") pod \"24c3bf4f-d669-4dbb-b217-ac508c47db91\" (UID: \"24c3bf4f-d669-4dbb-b217-ac508c47db91\") "
Oct 30 03:23:40.303559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.303532    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume" pod "24c3bf4f-d669-4dbb-b217-ac508c47db91" (UID: "24c3bf4f-d669-4dbb-b217-ac508c47db91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:40.305465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.305408    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj" pod "24c3bf4f-d669-4dbb-b217-ac508c47db91" (UID: "24c3bf4f-d669-4dbb-b217-ac508c47db91"). InnerVolumeSpecName "kube-api-access-2f9sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:40.404585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.404545    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c3bf4f-d669-4dbb-b217-ac508c47db91-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:40.404585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.404584    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9sj\" (UniqueName: \"kubernetes.io/projected/24c3bf4f-d669-4dbb-b217-ac508c47db91-kube-api-access-2f9sj\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:40.682897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.682862    4877 generic.go:356] "Generic (PLEG): container finished" podID="24c3bf4f-d669-4dbb-b217-ac508c47db91" containerID="b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394" exitCode=0
Oct 30 03:23:40.683087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.682923    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"
Oct 30 03:23:40.683087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.682935    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f" event={"ID":"24c3bf4f-d669-4dbb-b217-ac508c47db91","Type":"ContainerDied","Data":"b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394"}
Oct 30 03:23:40.683087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.683000    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f" event={"ID":"24c3bf4f-d669-4dbb-b217-ac508c47db91","Type":"ContainerDied","Data":"eeb82cee539c8b135131250dfefe5343804d3f1ecaf50a2074df75afa3478110"}
Oct 30 03:23:40.683087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.683029    4877 scope.go:122] "RemoveContainer" containerID="b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394"
Oct 30 03:23:40.699873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.699840    4877 scope.go:122] "RemoveContainer" containerID="b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394"
Oct 30 03:23:40.700482 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:23:40.700417    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394\": container with ID starting with b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394 not found: ID does not exist" containerID="b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394"
Oct 30 03:23:40.700598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.700487    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394"} err="failed to get container status \"b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394\": rpc error: code = NotFound desc = could not find container \"b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394\": container with ID starting with b1daddde36c838a6931140e89e253a08aa4691a3e258e8acf81088826d6b4394 not found: ID does not exist"
Oct 30 03:23:40.735296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.735265    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"]
Oct 30 03:23:40.747358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:40.747325    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-79d784d8bc-9tb2f"]
Oct 30 03:23:41.699617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:41.699581    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c3bf4f-d669-4dbb-b217-ac508c47db91" path="/var/lib/kubelet/pods/24c3bf4f-d669-4dbb-b217-ac508c47db91/volumes"
Oct 30 03:23:46.230821 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:23:46.230775    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:23:46.270639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:46.270605    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"]
Oct 30 03:23:49.675455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:49.675403    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:23:49.676041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:49.675488    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:23:51.271226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.271165    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7" podUID="76a24968-074a-401c-906c-788c25f64566" containerName="payload-pre-processing" containerID="cri-o://6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804" gracePeriod=25
Oct 30 03:23:51.563175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.563143    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:51.665528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.665487    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz\" (UniqueName: \"kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz\") pod \"76a24968-074a-401c-906c-788c25f64566\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") "
Oct 30 03:23:51.665728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.665614    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume\") pod \"76a24968-074a-401c-906c-788c25f64566\" (UID: \"76a24968-074a-401c-906c-788c25f64566\") "
Oct 30 03:23:51.666055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.666029    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume" pod "76a24968-074a-401c-906c-788c25f64566" (UID: "76a24968-074a-401c-906c-788c25f64566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:23:51.666158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.666143    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a24968-074a-401c-906c-788c25f64566-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:51.667631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.667602    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz" pod "76a24968-074a-401c-906c-788c25f64566" (UID: "76a24968-074a-401c-906c-788c25f64566"). InnerVolumeSpecName "kube-api-access-4twqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:51.767244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.767207    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4twqz\" (UniqueName: \"kubernetes.io/projected/76a24968-074a-401c-906c-788c25f64566-kube-api-access-4twqz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:23:51.776286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.776257    4877 generic.go:356] "Generic (PLEG): container finished" podID="76a24968-074a-401c-906c-788c25f64566" containerID="6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804" exitCode=0
Oct 30 03:23:51.776460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.776322    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"
Oct 30 03:23:51.776460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.776383    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7" event={"ID":"76a24968-074a-401c-906c-788c25f64566","Type":"ContainerDied","Data":"6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804"}
Oct 30 03:23:51.776556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.776460    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7" event={"ID":"76a24968-074a-401c-906c-788c25f64566","Type":"ContainerDied","Data":"f68e7269d1515079936e28d8d98b1191380f2095cbb8f31996bd86dd3acdebd8"}
Oct 30 03:23:51.776556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.776491    4877 scope.go:122] "RemoveContainer" containerID="6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804"
Oct 30 03:23:51.786826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.786809    4877 scope.go:122] "RemoveContainer" containerID="6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804"
Oct 30 03:23:51.787072 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:23:51.787054    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804\": container with ID starting with 6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804 not found: ID does not exist" containerID="6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804"
Oct 30 03:23:51.787141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.787080    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804"} err="failed to get container status \"6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804\": rpc error: code = NotFound desc = could not find container \"6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804\": container with ID starting with 6df6d4bb0529eb63dfc270030a573a7efbe865cb4a2635ce116833d2140b1804 not found: ID does not exist"
Oct 30 03:23:51.827945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.827915    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"]
Oct 30 03:23:51.833345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:51.833317    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-649b997699-gp2n7"]
Oct 30 03:23:52.034472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.034439    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"]
Oct 30 03:23:52.041505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.041437    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a3eaf3-46ccg"]
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.474768    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"]
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.475528    4877 cpu_manager.go:414] "containerMap: removing container" podUID="24c3bf4f-d669-4dbb-b217-ac508c47db91" containerName="payload-pre-processing"
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.475544    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c3bf4f-d669-4dbb-b217-ac508c47db91" containerName="payload-pre-processing"
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.475560    4877 cpu_manager.go:414] "containerMap: removing container" podUID="76a24968-074a-401c-906c-788c25f64566" containerName="payload-pre-processing"
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.475565    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a24968-074a-401c-906c-788c25f64566" containerName="payload-pre-processing"
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.475663    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c3bf4f-d669-4dbb-b217-ac508c47db91" containerName="payload-pre-processing"
Oct 30 03:23:52.476527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.475677    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a24968-074a-401c-906c-788c25f64566" containerName="payload-pre-processing"
Oct 30 03:23:52.482086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.482045    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.501353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.498560    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a1e5d5-dockercfg-kfmxs\""
Oct 30 03:23:52.501353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.498685    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1e5d5\""
Oct 30 03:23:52.505239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.505209    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"]
Oct 30 03:23:52.578497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.578466    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.578657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.578503    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxsn5\" (UniqueName: \"kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.639854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.639815    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"]
Oct 30 03:23:52.644096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.644073    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.676564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.676535    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"]
Oct 30 03:23:52.679770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.679738    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.679903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.679784    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxsn5\" (UniqueName: \"kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.680806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.680781    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.699020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.698988    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxsn5\" (UniqueName: \"kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5\") pod \"payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.780718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.780679    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.780907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.780800    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf99t\" (UniqueName: \"kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t\") pod \"payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.810096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.810067    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:52.888459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.883207    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.888459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.883320    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf99t\" (UniqueName: \"kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t\") pod \"payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.888459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.884694    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.903809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.903778    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf99t\" (UniqueName: \"kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t\") pod \"payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:52.959909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:52.959862    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:53.117012 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:53.116975    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77425bc0_2a1a_445f_8254_54e977a01b41.slice/crio-99ea6ade8c30d28dd6e0486a31e69fcb515a391ca72bd6b9fb2be961a4c2c45c WatchSource:0}: Error finding container 99ea6ade8c30d28dd6e0486a31e69fcb515a391ca72bd6b9fb2be961a4c2c45c: Status 404 returned error can't find the container with id 99ea6ade8c30d28dd6e0486a31e69fcb515a391ca72bd6b9fb2be961a4c2c45c
Oct 30 03:23:53.120032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.120001    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"]
Oct 30 03:23:53.269687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.269659    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"]
Oct 30 03:23:53.270035 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:53.270006    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16334de_f241_4ebb_8d14_8026394efc82.slice/crio-bf1d77459024c916f1f34678811416df7b63ad20c1bc80c3bf641be75fec5357 WatchSource:0}: Error finding container bf1d77459024c916f1f34678811416df7b63ad20c1bc80c3bf641be75fec5357: Status 404 returned error can't find the container with id bf1d77459024c916f1f34678811416df7b63ad20c1bc80c3bf641be75fec5357
Oct 30 03:23:53.703196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.703158    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd6ca6d-ab78-4c51-96f2-82c38b435c17" path="/var/lib/kubelet/pods/6dd6ca6d-ab78-4c51-96f2-82c38b435c17/volumes"
Oct 30 03:23:53.704338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.704315    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a24968-074a-401c-906c-788c25f64566" path="/var/lib/kubelet/pods/76a24968-074a-401c-906c-788c25f64566/volumes"
Oct 30 03:23:53.795629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.795575    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn" event={"ID":"b16334de-f241-4ebb-8d14-8026394efc82","Type":"ContainerStarted","Data":"063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6"}
Oct 30 03:23:53.795830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.795665    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn" event={"ID":"b16334de-f241-4ebb-8d14-8026394efc82","Type":"ContainerStarted","Data":"bf1d77459024c916f1f34678811416df7b63ad20c1bc80c3bf641be75fec5357"}
Oct 30 03:23:53.795830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.795716    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:23:53.797664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.797613    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2" event={"ID":"77425bc0-2a1a-445f-8254-54e977a01b41","Type":"ContainerStarted","Data":"d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45"}
Oct 30 03:23:53.797812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.797685    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2" event={"ID":"77425bc0-2a1a-445f-8254-54e977a01b41","Type":"ContainerStarted","Data":"99ea6ade8c30d28dd6e0486a31e69fcb515a391ca72bd6b9fb2be961a4c2c45c"}
Oct 30 03:23:53.797812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.797714    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:23:53.839965 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.839912    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn" podStartSLOduration=1.839891714 podStartE2EDuration="1.839891714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:53.83915226 +0000 UTC m=+1262.268041680" watchObservedRunningTime="2026-10-30 03:23:53.839891714 +0000 UTC m=+1262.268781131"
Oct 30 03:23:53.881481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:53.881403    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2" podStartSLOduration=1.881382061 podStartE2EDuration="1.881382061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:53.877268833 +0000 UTC m=+1262.306158274" watchObservedRunningTime="2026-10-30 03:23:53.881382061 +0000 UTC m=+1262.310271484"
Oct 30 03:23:59.340081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.340044    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"]
Oct 30 03:23:59.345823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.345799    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.375469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.372270    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b03a5e\""
Oct 30 03:23:59.375469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.372339    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b03a5e-dockercfg-5rwkl\""
Oct 30 03:23:59.378243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.378214    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"]
Oct 30 03:23:59.489452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.487161    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsx4w\" (UniqueName: \"kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w\") pod \"payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.489452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.487443    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.588888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.588851    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.589109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.588940    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsx4w\" (UniqueName: \"kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w\") pod \"payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.589626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.589603    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.608074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.608000    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsx4w\" (UniqueName: \"kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w\") pod \"payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.660808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.660763    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:23:59.968342 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:23:59.968312    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"]
Oct 30 03:23:59.968470 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:23:59.968341    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde555438_38ef_487f_88e2_dfb7e7459f0a.slice/crio-9de4132d3ff65c71b7c5d57e6d96f9fa988afcb931abd0564fd96fd6a080d030 WatchSource:0}: Error finding container 9de4132d3ff65c71b7c5d57e6d96f9fa988afcb931abd0564fd96fd6a080d030: Status 404 returned error can't find the container with id 9de4132d3ff65c71b7c5d57e6d96f9fa988afcb931abd0564fd96fd6a080d030
Oct 30 03:24:00.859763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:00.859719    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz" event={"ID":"de555438-38ef-487f-88e2-dfb7e7459f0a","Type":"ContainerStarted","Data":"914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879"}
Oct 30 03:24:00.860236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:00.859783    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:24:00.860236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:00.859802    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz" event={"ID":"de555438-38ef-487f-88e2-dfb7e7459f0a","Type":"ContainerStarted","Data":"9de4132d3ff65c71b7c5d57e6d96f9fa988afcb931abd0564fd96fd6a080d030"}
Oct 30 03:24:00.903021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:00.902952    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz" podStartSLOduration=1.902932863 podStartE2EDuration="1.902932863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:00.9011213 +0000 UTC m=+1269.330010710" watchObservedRunningTime="2026-10-30 03:24:00.902932863 +0000 UTC m=+1269.331822281"
Oct 30 03:24:01.149943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:01.149859    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"]
Oct 30 03:24:04.811498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:04.811465    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:24:04.811893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:04.811552    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:24:06.151107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.151040    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt" podUID="031b3817-c93f-4318-b662-2cea5fff7c71" containerName="payload-processing" containerID="cri-o://204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb" gracePeriod=25
Oct 30 03:24:06.447989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.447954    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:24:06.473239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.473212    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume\") pod \"031b3817-c93f-4318-b662-2cea5fff7c71\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") "
Oct 30 03:24:06.473379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.473254    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q\" (UniqueName: \"kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q\") pod \"031b3817-c93f-4318-b662-2cea5fff7c71\" (UID: \"031b3817-c93f-4318-b662-2cea5fff7c71\") "
Oct 30 03:24:06.473670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.473647    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume" pod "031b3817-c93f-4318-b662-2cea5fff7c71" (UID: "031b3817-c93f-4318-b662-2cea5fff7c71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:06.473792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.473779    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/031b3817-c93f-4318-b662-2cea5fff7c71-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:06.475283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.475257    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q" pod "031b3817-c93f-4318-b662-2cea5fff7c71" (UID: "031b3817-c93f-4318-b662-2cea5fff7c71"). InnerVolumeSpecName "kube-api-access-45q2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:06.575115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.575075    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45q2q\" (UniqueName: \"kubernetes.io/projected/031b3817-c93f-4318-b662-2cea5fff7c71-kube-api-access-45q2q\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:06.912489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.912453    4877 generic.go:356] "Generic (PLEG): container finished" podID="031b3817-c93f-4318-b662-2cea5fff7c71" containerID="204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb" exitCode=0
Oct 30 03:24:06.912664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.912490    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt" event={"ID":"031b3817-c93f-4318-b662-2cea5fff7c71","Type":"ContainerDied","Data":"204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb"}
Oct 30 03:24:06.912664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.912520    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"
Oct 30 03:24:06.912664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.912562    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt" event={"ID":"031b3817-c93f-4318-b662-2cea5fff7c71","Type":"ContainerDied","Data":"1c241bb8c6d0f360816f651b98fa123965b34296050cf6f419f7e9515179be17"}
Oct 30 03:24:06.912664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.912593    4877 scope.go:122] "RemoveContainer" containerID="204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb"
Oct 30 03:24:06.924000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.923980    4877 scope.go:122] "RemoveContainer" containerID="204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb"
Oct 30 03:24:06.924290 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:06.924271    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb\": container with ID starting with 204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb not found: ID does not exist" containerID="204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb"
Oct 30 03:24:06.924364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.924301    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb"} err="failed to get container status \"204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb\": rpc error: code = NotFound desc = could not find container \"204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb\": container with ID starting with 204958e56e2197ee168e0fd45d73498b70211b4286fff18b68d550689d9ccafb not found: ID does not exist"
Oct 30 03:24:06.963506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.963478    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"]
Oct 30 03:24:06.969406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:06.969383    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-5f8c76fffd-sfzlt"]
Oct 30 03:24:07.702360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:07.702323    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031b3817-c93f-4318-b662-2cea5fff7c71" path="/var/lib/kubelet/pods/031b3817-c93f-4318-b662-2cea5fff7c71/volumes"
Oct 30 03:24:08.614144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.609444    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"]
Oct 30 03:24:08.614144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.610088    4877 cpu_manager.go:414] "containerMap: removing container" podUID="031b3817-c93f-4318-b662-2cea5fff7c71" containerName="payload-processing"
Oct 30 03:24:08.614144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.610103    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031b3817-c93f-4318-b662-2cea5fff7c71" containerName="payload-processing"
Oct 30 03:24:08.614144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.610376    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="031b3817-c93f-4318-b662-2cea5fff7c71" containerName="payload-processing"
Oct 30 03:24:08.615986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.615965    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.635542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.635510    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"]
Oct 30 03:24:08.641355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.641327    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6c106c\""
Oct 30 03:24:08.641578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.641506    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6c106c-dockercfg-gnnzn\""
Oct 30 03:24:08.715866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.715832    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.716353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.715890    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.774229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.774196    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"]
Oct 30 03:24:08.778681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.778651    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:08.798068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.798038    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"]
Oct 30 03:24:08.816999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.816966    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.817184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.817162    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zblt\" (UniqueName: \"kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt\") pod \"payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:08.817452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.817402    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.818398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.818376    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:08.818508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.818297    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.836076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.836045    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74\") pod \"payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.919911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.919828    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:08.920056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.919935    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zblt\" (UniqueName: \"kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt\") pod \"payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:08.921088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.921056    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:08.934216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.934188    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:08.939708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:08.939683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zblt\" (UniqueName: \"kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt\") pod \"payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:09.092751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.092724    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:09.240208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.240166    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"]
Oct 30 03:24:09.240629 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:24:09.240605    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada11452_b43e_4475_ab9e_321aa368f965.slice/crio-f72f1feff32fb52dc9542edcafba9a92356ab64b6579aee6d8faa418bfff36d2 WatchSource:0}: Error finding container f72f1feff32fb52dc9542edcafba9a92356ab64b6579aee6d8faa418bfff36d2: Status 404 returned error can't find the container with id f72f1feff32fb52dc9542edcafba9a92356ab64b6579aee6d8faa418bfff36d2
Oct 30 03:24:09.401956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.401916    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"]
Oct 30 03:24:09.947541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.947505    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" event={"ID":"ada11452-b43e-4475-ab9e-321aa368f965","Type":"ContainerStarted","Data":"21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29"}
Oct 30 03:24:09.948040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.947558    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" event={"ID":"ada11452-b43e-4475-ab9e-321aa368f965","Type":"ContainerStarted","Data":"f72f1feff32fb52dc9542edcafba9a92356ab64b6579aee6d8faa418bfff36d2"}
Oct 30 03:24:09.948040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.947615    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:09.950008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.949977    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" event={"ID":"3ad05ccb-8d11-4718-826b-5d05bcabe7a1","Type":"ContainerStarted","Data":"7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1"}
Oct 30 03:24:09.950229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.950204    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:09.950328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.950244    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" event={"ID":"3ad05ccb-8d11-4718-826b-5d05bcabe7a1","Type":"ContainerStarted","Data":"5f624f5a98ca55385b516aae4a3b9348c10970c64bdf39d0e22f4f55c240198a"}
Oct 30 03:24:09.993234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:09.993171    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" podStartSLOduration=1.9931537179999999 podStartE2EDuration="1.993153718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:09.988318136 +0000 UTC m=+1278.417207584" watchObservedRunningTime="2026-10-30 03:24:09.993153718 +0000 UTC m=+1278.422043140"
Oct 30 03:24:10.029578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:10.029529    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" podStartSLOduration=2.029514508 podStartE2EDuration="2.029514508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:10.025839511 +0000 UTC m=+1278.454728940" watchObservedRunningTime="2026-10-30 03:24:10.029514508 +0000 UTC m=+1278.458403926"
Oct 30 03:24:11.870485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:11.870449    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:24:16.229796 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:16.229751    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:24:20.961684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:20.961650    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:24:20.962239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:20.961744    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:24:21.179278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.179244    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"]
Oct 30 03:24:21.184894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.184866    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.207449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.204883    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0f003-dockercfg-xb8r7\""
Oct 30 03:24:21.207823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.207793    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0f003\""
Oct 30 03:24:21.221323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.221127    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"]
Oct 30 03:24:21.287192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.287135    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.287436 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.287253    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz\") pod \"payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.389443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.389393    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.389648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.389492    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz\") pod \"payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.390370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.390340    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.408764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.408731    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz\") pod \"payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.517071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.516981    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:21.837885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:21.826493    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"]
Oct 30 03:24:22.044181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:22.044139    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd" event={"ID":"d4ae5348-7992-40be-9d82-b543821d387d","Type":"ContainerStarted","Data":"f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223"}
Oct 30 03:24:22.044705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:22.044205    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd" event={"ID":"d4ae5348-7992-40be-9d82-b543821d387d","Type":"ContainerStarted","Data":"c0000bc9beb7d147d20e1285f00574a5da5c2710f647d3c849e73da56c123e06"}
Oct 30 03:24:22.044705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:22.044347    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:22.087499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:22.087369    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd" podStartSLOduration=1.087346768 podStartE2EDuration="1.087346768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:22.081935776 +0000 UTC m=+1290.510825219" watchObservedRunningTime="2026-10-30 03:24:22.087346768 +0000 UTC m=+1290.516236191"
Oct 30 03:24:33.053544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:33.053507    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:24:36.049000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.048907    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"]
Oct 30 03:24:36.049477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.049179    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx" podUID="0be95dad-ad7f-4b82-93be-f58082f057db" containerName="payload-pre-processing" containerID="cri-o://0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3" gracePeriod=30
Oct 30 03:24:36.430924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.430881    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:24:36.487509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.487470    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f\" (UniqueName: \"kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f\") pod \"0be95dad-ad7f-4b82-93be-f58082f057db\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") "
Oct 30 03:24:36.490522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.490483    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f" pod "0be95dad-ad7f-4b82-93be-f58082f057db" (UID: "0be95dad-ad7f-4b82-93be-f58082f057db"). InnerVolumeSpecName "kube-api-access-mtc6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:36.588986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.588900    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume\") pod \"0be95dad-ad7f-4b82-93be-f58082f057db\" (UID: \"0be95dad-ad7f-4b82-93be-f58082f057db\") "
Oct 30 03:24:36.589251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.589236    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtc6f\" (UniqueName: \"kubernetes.io/projected/0be95dad-ad7f-4b82-93be-f58082f057db-kube-api-access-mtc6f\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:36.589389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.589359    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume" pod "0be95dad-ad7f-4b82-93be-f58082f057db" (UID: "0be95dad-ad7f-4b82-93be-f58082f057db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:36.690602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:36.690569    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0be95dad-ad7f-4b82-93be-f58082f057db-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:37.167862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.167818    4877 generic.go:356] "Generic (PLEG): container finished" podID="0be95dad-ad7f-4b82-93be-f58082f057db" containerID="0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3" exitCode=0
Oct 30 03:24:37.168339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.167857    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx" event={"ID":"0be95dad-ad7f-4b82-93be-f58082f057db","Type":"ContainerDied","Data":"0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3"}
Oct 30 03:24:37.168339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.167935    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx" event={"ID":"0be95dad-ad7f-4b82-93be-f58082f057db","Type":"ContainerDied","Data":"d8ef691fe574b8245f50e58b07e35667ac1ec7b3d65cb7819a2dfea8f69c9053"}
Oct 30 03:24:37.168339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.167871    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"
Oct 30 03:24:37.168339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.167967    4877 scope.go:122] "RemoveContainer" containerID="0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3"
Oct 30 03:24:37.181602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.181580    4877 scope.go:122] "RemoveContainer" containerID="0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3"
Oct 30 03:24:37.181934 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:37.181902    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3\": container with ID starting with 0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3 not found: ID does not exist" containerID="0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3"
Oct 30 03:24:37.182004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.181949    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3"} err="failed to get container status \"0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3\": rpc error: code = NotFound desc = could not find container \"0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3\": container with ID starting with 0890f13a6987612f07d12637df0fb3cf4cf574741e0bb82dc5b0c9950b5774e3 not found: ID does not exist"
Oct 30 03:24:37.220467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.220418    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"]
Oct 30 03:24:37.231683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.231651    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-6cd8f8f567-8kdjx"]
Oct 30 03:24:37.699834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:37.699797    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be95dad-ad7f-4b82-93be-f58082f057db" path="/var/lib/kubelet/pods/0be95dad-ad7f-4b82-93be-f58082f057db/volumes"
Oct 30 03:24:40.819823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:40.819702    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"]
Oct 30 03:24:40.820645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:40.820606    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy" containerID="cri-o://c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe" gracePeriod=30
Oct 30 03:24:46.123738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.123704    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:24:46.234767 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:46.234721    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:24:46.241620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.241587    4877 generic.go:356] "Generic (PLEG): container finished" podID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerID="c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe" exitCode=0
Oct 30 03:24:46.241791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.241631    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" event={"ID":"4e3b31c2-a4df-4aed-b955-0639253f104c","Type":"ContainerDied","Data":"c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe"}
Oct 30 03:24:46.241791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.241640    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"
Oct 30 03:24:46.241791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.241678    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx" event={"ID":"4e3b31c2-a4df-4aed-b955-0639253f104c","Type":"ContainerDied","Data":"0ed62f70bd2f315cd31220ee9ed62e330f0065d6a8619a27b0a3e27d66dfec3f"}
Oct 30 03:24:46.241791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.241705    4877 scope.go:122] "RemoveContainer" containerID="c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe"
Oct 30 03:24:46.245120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245092    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.245235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245165    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.245308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245270    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.245365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245307    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.245365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245337    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.245520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245469    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.245579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245564    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.246511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245622    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.246511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245665    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:46.246511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.245746    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv\") pod \"4e3b31c2-a4df-4aed-b955-0639253f104c\" (UID: \"4e3b31c2-a4df-4aed-b955-0639253f104c\") "
Oct 30 03:24:46.246511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.246192    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b31c2-a4df-4aed-b955-0639253f104c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.246511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.246206    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:46.246843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.246650    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:46.246843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.246731    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:46.246981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.246948    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:46.247705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.247680    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:46.247952 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.247933    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:46.248915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.248892    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:46.249784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.249757    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv" pod "4e3b31c2-a4df-4aed-b955-0639253f104c" (UID: "4e3b31c2-a4df-4aed-b955-0639253f104c"). InnerVolumeSpecName "kube-api-access-pbkrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:46.254909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.254885    4877 scope.go:122] "RemoveContainer" containerID="0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"
Oct 30 03:24:46.278946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.278922    4877 scope.go:122] "RemoveContainer" containerID="c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe"
Oct 30 03:24:46.279362 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:46.279330    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe\": container with ID starting with c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe not found: ID does not exist" containerID="c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe"
Oct 30 03:24:46.279405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.279374    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe"} err="failed to get container status \"c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe\": rpc error: code = NotFound desc = could not find container \"c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe\": container with ID starting with c60f7674ccc65f4194c388aee5774f0ad374f843f91313cd7f73b9e707ffe6fe not found: ID does not exist"
Oct 30 03:24:46.279405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.279394    4877 scope.go:122] "RemoveContainer" containerID="0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"
Oct 30 03:24:46.279659 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:46.279641    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c\": container with ID starting with 0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c not found: ID does not exist" containerID="0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"
Oct 30 03:24:46.279710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.279663    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c"} err="failed to get container status \"0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c\": rpc error: code = NotFound desc = could not find container \"0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c\": container with ID starting with 0b997b1f1fd16fb08ba4be19d0e7761c4f82da9f1c10469f46d10b48deef334c not found: ID does not exist"
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347502    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkrv\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-kube-api-access-pbkrv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347536    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347549    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347561    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347573    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347586    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347599    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.347944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.347611    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b31c2-a4df-4aed-b955-0639253f104c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:46.587834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.587794    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"]
Oct 30 03:24:46.599852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:46.599777    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d6c536-openshift-default-5fcfcf5b5-wq4tx"]
Oct 30 03:24:47.701072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:47.701033    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" path="/var/lib/kubelet/pods/4e3b31c2-a4df-4aed-b955-0639253f104c/volumes"
Oct 30 03:24:50.076181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.076146    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"]
Oct 30 03:24:50.076651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.076433    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq" podUID="502500f2-9ce1-47e9-88f7-1649164bd142" containerName="payload-processing" containerID="cri-o://9a2c72bd6ef946701ceb18cb03c4c75b0ffbc71540fe52e2cbbc7adb4ead60d3" gracePeriod=30
Oct 30 03:24:50.100969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.100932    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"]
Oct 30 03:24:50.101244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.101196    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv" podUID="3c7cb16e-37e3-4463-877d-efae398806f0" containerName="payload-pre-processing" containerID="cri-o://7d2f10cb3613a6cc380a0fc16c27cb39d4e3bbd091058e3edf229b1a017686fe" gracePeriod=30
Oct 30 03:24:50.277482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.277271    4877 generic.go:356] "Generic (PLEG): container finished" podID="502500f2-9ce1-47e9-88f7-1649164bd142" containerID="9a2c72bd6ef946701ceb18cb03c4c75b0ffbc71540fe52e2cbbc7adb4ead60d3" exitCode=0
Oct 30 03:24:50.277482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.277398    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq" event={"ID":"502500f2-9ce1-47e9-88f7-1649164bd142","Type":"ContainerDied","Data":"9a2c72bd6ef946701ceb18cb03c4c75b0ffbc71540fe52e2cbbc7adb4ead60d3"}
Oct 30 03:24:50.279274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.279246    4877 generic.go:356] "Generic (PLEG): container finished" podID="3c7cb16e-37e3-4463-877d-efae398806f0" containerID="7d2f10cb3613a6cc380a0fc16c27cb39d4e3bbd091058e3edf229b1a017686fe" exitCode=0
Oct 30 03:24:50.279414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.279382    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv" event={"ID":"3c7cb16e-37e3-4463-877d-efae398806f0","Type":"ContainerDied","Data":"7d2f10cb3613a6cc380a0fc16c27cb39d4e3bbd091058e3edf229b1a017686fe"}
Oct 30 03:24:50.506510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.502651    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:24:50.524662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.524630    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:24:50.615738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.611781    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm\" (UniqueName: \"kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm\") pod \"3c7cb16e-37e3-4463-877d-efae398806f0\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") "
Oct 30 03:24:50.615738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.611863    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume\") pod \"3c7cb16e-37e3-4463-877d-efae398806f0\" (UID: \"3c7cb16e-37e3-4463-877d-efae398806f0\") "
Oct 30 03:24:50.615738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.612078    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx\" (UniqueName: \"kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx\") pod \"502500f2-9ce1-47e9-88f7-1649164bd142\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") "
Oct 30 03:24:50.615738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.612181    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume\") pod \"502500f2-9ce1-47e9-88f7-1649164bd142\" (UID: \"502500f2-9ce1-47e9-88f7-1649164bd142\") "
Oct 30 03:24:50.615738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.615202    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume" pod "502500f2-9ce1-47e9-88f7-1649164bd142" (UID: "502500f2-9ce1-47e9-88f7-1649164bd142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:50.615738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.615630    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume" pod "3c7cb16e-37e3-4463-877d-efae398806f0" (UID: "3c7cb16e-37e3-4463-877d-efae398806f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:50.619499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.619449    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm" pod "3c7cb16e-37e3-4463-877d-efae398806f0" (UID: "3c7cb16e-37e3-4463-877d-efae398806f0"). InnerVolumeSpecName "kube-api-access-zrmkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:50.621774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.621731    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx" pod "502500f2-9ce1-47e9-88f7-1649164bd142" (UID: "502500f2-9ce1-47e9-88f7-1649164bd142"). InnerVolumeSpecName "kube-api-access-78lrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:50.715470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.715414    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmkm\" (UniqueName: \"kubernetes.io/projected/3c7cb16e-37e3-4463-877d-efae398806f0-kube-api-access-zrmkm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:50.715470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.715457    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7cb16e-37e3-4463-877d-efae398806f0-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:50.715470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.715468    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lrx\" (UniqueName: \"kubernetes.io/projected/502500f2-9ce1-47e9-88f7-1649164bd142-kube-api-access-78lrx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:50.715470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:50.715478    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/502500f2-9ce1-47e9-88f7-1649164bd142-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:51.288821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.288793    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"
Oct 30 03:24:51.289311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.288825    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv" event={"ID":"3c7cb16e-37e3-4463-877d-efae398806f0","Type":"ContainerDied","Data":"fff85a356658c016ab001b9dab6b0975af2ce670bd089142aa5a4225de3919a6"}
Oct 30 03:24:51.289311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.288894    4877 scope.go:122] "RemoveContainer" containerID="7d2f10cb3613a6cc380a0fc16c27cb39d4e3bbd091058e3edf229b1a017686fe"
Oct 30 03:24:51.290756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.290732    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"
Oct 30 03:24:51.290859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.290739    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq" event={"ID":"502500f2-9ce1-47e9-88f7-1649164bd142","Type":"ContainerDied","Data":"41847bdc805dca048fd9a6a5beff008e5cd60d50027e69d1a482607950a795c2"}
Oct 30 03:24:51.300839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.300817    4877 scope.go:122] "RemoveContainer" containerID="9a2c72bd6ef946701ceb18cb03c4c75b0ffbc71540fe52e2cbbc7adb4ead60d3"
Oct 30 03:24:51.343192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.343156    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"]
Oct 30 03:24:51.353199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.353106    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-654b554d84-nz2qq"]
Oct 30 03:24:51.379760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.379729    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"]
Oct 30 03:24:51.395571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.393523    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04fdf3-cdc8f8747-hqvkv"]
Oct 30 03:24:51.703184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.703101    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7cb16e-37e3-4463-877d-efae398806f0" path="/var/lib/kubelet/pods/3c7cb16e-37e3-4463-877d-efae398806f0/volumes"
Oct 30 03:24:51.703767 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:51.703744    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502500f2-9ce1-47e9-88f7-1649164bd142" path="/var/lib/kubelet/pods/502500f2-9ce1-47e9-88f7-1649164bd142/volumes"
Oct 30 03:24:53.578647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:53.578607    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"]
Oct 30 03:24:53.579664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:53.578885    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy" containerID="cri-o://59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89" gracePeriod=30
Oct 30 03:24:53.667586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:53.667549    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"]
Oct 30 03:24:54.285588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:54.285559    4877 scope.go:122] "RemoveContainer" containerID="95303021abfabe005a4963b2cd5beb688e46a67cd3ada0937f4aa5d842ae297e"
Oct 30 03:24:54.302071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:54.302044    4877 scope.go:122] "RemoveContainer" containerID="417725000f9a430c42a4f693e9ed42d8d5de5e44af35b2437f64804960bcb815"
Oct 30 03:24:58.668121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:58.668041    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz" podUID="de555438-38ef-487f-88e2-dfb7e7459f0a" containerName="payload-processing" containerID="cri-o://914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879" gracePeriod=25
Oct 30 03:24:58.965385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:58.965356    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:24:59.005517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.005485    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:24:59.134821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.134783    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume\") pod \"de555438-38ef-487f-88e2-dfb7e7459f0a\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") "
Oct 30 03:24:59.135008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.134843    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.134971    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135073    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135137    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135200    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135245    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume" pod "de555438-38ef-487f-88e2-dfb7e7459f0a" (UID: "de555438-38ef-487f-88e2-dfb7e7459f0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:59.135362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135259    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135301    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135363    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135369    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:59.135537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135457    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert\") pod \"075ae435-693b-406e-ad16-3d253430068b\" (UID: \"075ae435-693b-406e-ad16-3d253430068b\") "
Oct 30 03:24:59.135537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135508    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w\" (UniqueName: \"kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w\") pod \"de555438-38ef-487f-88e2-dfb7e7459f0a\" (UID: \"de555438-38ef-487f-88e2-dfb7e7459f0a\") "
Oct 30 03:24:59.135755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135613    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:59.135826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.135802    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:59.136154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136108    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:59.136270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136181    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:59.136356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136324    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.136356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136348    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.136543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136363    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.136543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136378    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.136543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136393    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075ae435-693b-406e-ad16-3d253430068b-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.136543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.136408    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de555438-38ef-487f-88e2-dfb7e7459f0a-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.137515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.137479    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:59.137515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.137502    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:59.137744 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.137727    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "kube-api-access-tlzld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:59.138177 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.138156    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo" pod "075ae435-693b-406e-ad16-3d253430068b" (UID: "075ae435-693b-406e-ad16-3d253430068b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:59.138477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.138452    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w" pod "de555438-38ef-487f-88e2-dfb7e7459f0a" (UID: "de555438-38ef-487f-88e2-dfb7e7459f0a"). InnerVolumeSpecName "kube-api-access-gsx4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:59.237580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.237487    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075ae435-693b-406e-ad16-3d253430068b-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.237580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.237531    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075ae435-693b-406e-ad16-3d253430068b-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.237580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.237547    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsx4w\" (UniqueName: \"kubernetes.io/projected/de555438-38ef-487f-88e2-dfb7e7459f0a-kube-api-access-gsx4w\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.237580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.237564    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.237580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.237578    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzld\" (UniqueName: \"kubernetes.io/projected/075ae435-693b-406e-ad16-3d253430068b-kube-api-access-tlzld\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:24:59.355552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.355515    4877 generic.go:356] "Generic (PLEG): container finished" podID="075ae435-693b-406e-ad16-3d253430068b" containerID="59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89" exitCode=0
Oct 30 03:24:59.355752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.355585    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"
Oct 30 03:24:59.355752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.355598    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" event={"ID":"075ae435-693b-406e-ad16-3d253430068b","Type":"ContainerDied","Data":"59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89"}
Oct 30 03:24:59.355752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.355686    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd" event={"ID":"075ae435-693b-406e-ad16-3d253430068b","Type":"ContainerDied","Data":"229a23f720a34f43b8ebf7c6dd9d0fda6a0334d795b0cb6b782ddf6be251104e"}
Oct 30 03:24:59.355752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.355714    4877 scope.go:122] "RemoveContainer" containerID="59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89"
Oct 30 03:24:59.357089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.357066    4877 generic.go:356] "Generic (PLEG): container finished" podID="de555438-38ef-487f-88e2-dfb7e7459f0a" containerID="914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879" exitCode=0
Oct 30 03:24:59.357195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.357121    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz" event={"ID":"de555438-38ef-487f-88e2-dfb7e7459f0a","Type":"ContainerDied","Data":"914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879"}
Oct 30 03:24:59.357195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.357163    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz" event={"ID":"de555438-38ef-487f-88e2-dfb7e7459f0a","Type":"ContainerDied","Data":"9de4132d3ff65c71b7c5d57e6d96f9fa988afcb931abd0564fd96fd6a080d030"}
Oct 30 03:24:59.357315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.357229    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"
Oct 30 03:24:59.367056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.367036    4877 scope.go:122] "RemoveContainer" containerID="b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"
Oct 30 03:24:59.381494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.381404    4877 scope.go:122] "RemoveContainer" containerID="59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89"
Oct 30 03:24:59.381933 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:59.381900    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89\": container with ID starting with 59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89 not found: ID does not exist" containerID="59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89"
Oct 30 03:24:59.382042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.381943    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89"} err="failed to get container status \"59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89\": rpc error: code = NotFound desc = could not find container \"59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89\": container with ID starting with 59d618291aff244e193a865129acd1d0706921e1b1cd7bfaf8a6998938157e89 not found: ID does not exist"
Oct 30 03:24:59.382042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.381971    4877 scope.go:122] "RemoveContainer" containerID="b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"
Oct 30 03:24:59.382292 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:59.382272    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1\": container with ID starting with b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1 not found: ID does not exist" containerID="b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"
Oct 30 03:24:59.382344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.382299    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1"} err="failed to get container status \"b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1\": rpc error: code = NotFound desc = could not find container \"b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1\": container with ID starting with b8e4a5c67f6c03a4c3def19b7028503372b2287059cfca28f5a8ea40c01450c1 not found: ID does not exist"
Oct 30 03:24:59.382344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.382317    4877 scope.go:122] "RemoveContainer" containerID="914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879"
Oct 30 03:24:59.394462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.394410    4877 scope.go:122] "RemoveContainer" containerID="914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879"
Oct 30 03:24:59.394801 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:24:59.394784    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879\": container with ID starting with 914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879 not found: ID does not exist" containerID="914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879"
Oct 30 03:24:59.394863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.394807    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879"} err="failed to get container status \"914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879\": rpc error: code = NotFound desc = could not find container \"914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879\": container with ID starting with 914243807efce3555bf14773acb25fd07b2f7b93422682f43652350ba9c34879 not found: ID does not exist"
Oct 30 03:24:59.420169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.420137    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"]
Oct 30 03:24:59.423657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.423629    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-04fdf3-openshift-default-c7b5c969c-5tnkd"]
Oct 30 03:24:59.457716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.457681    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"]
Oct 30 03:24:59.461087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.461061    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-5786848cf8-nkkqz"]
Oct 30 03:24:59.701275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.701244    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075ae435-693b-406e-ad16-3d253430068b" path="/var/lib/kubelet/pods/075ae435-693b-406e-ad16-3d253430068b/volumes"
Oct 30 03:24:59.701718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:24:59.701702    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de555438-38ef-487f-88e2-dfb7e7459f0a" path="/var/lib/kubelet/pods/de555438-38ef-487f-88e2-dfb7e7459f0a/volumes"
Oct 30 03:25:05.821721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:05.821675    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"]
Oct 30 03:25:05.822246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:05.821886    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn" podUID="b16334de-f241-4ebb-8d14-8026394efc82" containerName="payload-processing" containerID="cri-o://063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6" gracePeriod=30
Oct 30 03:25:05.853199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:05.853168    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"]
Oct 30 03:25:05.853446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:05.853408    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2" podUID="77425bc0-2a1a-445f-8254-54e977a01b41" containerName="payload-pre-processing" containerID="cri-o://d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45" gracePeriod=30
Oct 30 03:25:06.226252 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.226212    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:25:06.245958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.245926    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:25:06.337597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.337502    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume\") pod \"b16334de-f241-4ebb-8d14-8026394efc82\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") "
Oct 30 03:25:06.337782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.337723    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5\" (UniqueName: \"kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5\") pod \"77425bc0-2a1a-445f-8254-54e977a01b41\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") "
Oct 30 03:25:06.337898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.337863    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t\" (UniqueName: \"kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t\") pod \"b16334de-f241-4ebb-8d14-8026394efc82\" (UID: \"b16334de-f241-4ebb-8d14-8026394efc82\") "
Oct 30 03:25:06.337969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.337908    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume\") pod \"77425bc0-2a1a-445f-8254-54e977a01b41\" (UID: \"77425bc0-2a1a-445f-8254-54e977a01b41\") "
Oct 30 03:25:06.338033 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.337974    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume" pod "b16334de-f241-4ebb-8d14-8026394efc82" (UID: "b16334de-f241-4ebb-8d14-8026394efc82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:06.338389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.338362    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16334de-f241-4ebb-8d14-8026394efc82-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:25:06.338572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.338543    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume" pod "77425bc0-2a1a-445f-8254-54e977a01b41" (UID: "77425bc0-2a1a-445f-8254-54e977a01b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:06.339975 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.339951    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t" pod "b16334de-f241-4ebb-8d14-8026394efc82" (UID: "b16334de-f241-4ebb-8d14-8026394efc82"). InnerVolumeSpecName "kube-api-access-jf99t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:06.340568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.340543    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5" pod "77425bc0-2a1a-445f-8254-54e977a01b41" (UID: "77425bc0-2a1a-445f-8254-54e977a01b41"). InnerVolumeSpecName "kube-api-access-vxsn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:06.422844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.422807    4877 generic.go:356] "Generic (PLEG): container finished" podID="77425bc0-2a1a-445f-8254-54e977a01b41" containerID="d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45" exitCode=0
Oct 30 03:25:06.423015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.422888    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"
Oct 30 03:25:06.423015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.422886    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2" event={"ID":"77425bc0-2a1a-445f-8254-54e977a01b41","Type":"ContainerDied","Data":"d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45"}
Oct 30 03:25:06.423141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.423077    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2" event={"ID":"77425bc0-2a1a-445f-8254-54e977a01b41","Type":"ContainerDied","Data":"99ea6ade8c30d28dd6e0486a31e69fcb515a391ca72bd6b9fb2be961a4c2c45c"}
Oct 30 03:25:06.423141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.423116    4877 scope.go:122] "RemoveContainer" containerID="d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45"
Oct 30 03:25:06.425075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.424948    4877 generic.go:356] "Generic (PLEG): container finished" podID="b16334de-f241-4ebb-8d14-8026394efc82" containerID="063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6" exitCode=0
Oct 30 03:25:06.425075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.424976    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn" event={"ID":"b16334de-f241-4ebb-8d14-8026394efc82","Type":"ContainerDied","Data":"063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6"}
Oct 30 03:25:06.425075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.425012    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"
Oct 30 03:25:06.425075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.425027    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn" event={"ID":"b16334de-f241-4ebb-8d14-8026394efc82","Type":"ContainerDied","Data":"bf1d77459024c916f1f34678811416df7b63ad20c1bc80c3bf641be75fec5357"}
Oct 30 03:25:06.434103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.434082    4877 scope.go:122] "RemoveContainer" containerID="d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45"
Oct 30 03:25:06.434358 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:25:06.434340    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45\": container with ID starting with d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45 not found: ID does not exist" containerID="d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45"
Oct 30 03:25:06.434469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.434366    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45"} err="failed to get container status \"d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45\": rpc error: code = NotFound desc = could not find container \"d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45\": container with ID starting with d95b819289bfe52d51003db25023f5b7e2d85730da174693b3fa52bafe822b45 not found: ID does not exist"
Oct 30 03:25:06.434469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.434383    4877 scope.go:122] "RemoveContainer" containerID="063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6"
Oct 30 03:25:06.439606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.439585    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxsn5\" (UniqueName: \"kubernetes.io/projected/77425bc0-2a1a-445f-8254-54e977a01b41-kube-api-access-vxsn5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:25:06.439606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.439607    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf99t\" (UniqueName: \"kubernetes.io/projected/b16334de-f241-4ebb-8d14-8026394efc82-kube-api-access-jf99t\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:25:06.439727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.439616    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77425bc0-2a1a-445f-8254-54e977a01b41-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:25:06.444540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.444522    4877 scope.go:122] "RemoveContainer" containerID="063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6"
Oct 30 03:25:06.444836 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:25:06.444815    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6\": container with ID starting with 063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6 not found: ID does not exist" containerID="063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6"
Oct 30 03:25:06.444912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.444856    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6"} err="failed to get container status \"063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6\": rpc error: code = NotFound desc = could not find container \"063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6\": container with ID starting with 063449a316137ba2563c01c8e3a78dce63c71cac6e28c19ba5cf34348a8decd6 not found: ID does not exist"
Oct 30 03:25:06.473122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.473094    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"]
Oct 30 03:25:06.476453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.476408    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1e5d5-566c96f799-wtvb2"]
Oct 30 03:25:06.509623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.509593    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"]
Oct 30 03:25:06.514898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:06.514872    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-798584d5c8-k8xcn"]
Oct 30 03:25:07.699705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:07.699674    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77425bc0-2a1a-445f-8254-54e977a01b41" path="/var/lib/kubelet/pods/77425bc0-2a1a-445f-8254-54e977a01b41/volumes"
Oct 30 03:25:07.700128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:07.700113    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16334de-f241-4ebb-8d14-8026394efc82" path="/var/lib/kubelet/pods/b16334de-f241-4ebb-8d14-8026394efc82/volumes"
Oct 30 03:25:10.089526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.089474    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"]
Oct 30 03:25:10.089990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.089736    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd" podUID="d4ae5348-7992-40be-9d82-b543821d387d" containerName="payload-processing" containerID="cri-o://f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223" gracePeriod=30
Oct 30 03:25:10.370533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.370506    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:25:10.478134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.478102    4877 generic.go:356] "Generic (PLEG): container finished" podID="d4ae5348-7992-40be-9d82-b543821d387d" containerID="f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223" exitCode=0
Oct 30 03:25:10.478546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.478525    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd" event={"ID":"d4ae5348-7992-40be-9d82-b543821d387d","Type":"ContainerDied","Data":"f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223"}
Oct 30 03:25:10.478690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.478616    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd" event={"ID":"d4ae5348-7992-40be-9d82-b543821d387d","Type":"ContainerDied","Data":"c0000bc9beb7d147d20e1285f00574a5da5c2710f647d3c849e73da56c123e06"}
Oct 30 03:25:10.478690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.478653    4877 scope.go:122] "RemoveContainer" containerID="f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223"
Oct 30 03:25:10.478881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.478799    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"
Oct 30 03:25:10.493667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.493646    4877 scope.go:122] "RemoveContainer" containerID="f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223"
Oct 30 03:25:10.496560 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:25:10.496527    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223\": container with ID starting with f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223 not found: ID does not exist" containerID="f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223"
Oct 30 03:25:10.496677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.496571    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223"} err="failed to get container status \"f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223\": rpc error: code = NotFound desc = could not find container \"f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223\": container with ID starting with f3eac72b106e22aca750583506cec550d0523d5e0d0b5fa0f0ca8dd217260223 not found: ID does not exist"
Oct 30 03:25:10.508021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.507993    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz\") pod \"d4ae5348-7992-40be-9d82-b543821d387d\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") "
Oct 30 03:25:10.508117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.508028    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume\") pod \"d4ae5348-7992-40be-9d82-b543821d387d\" (UID: \"d4ae5348-7992-40be-9d82-b543821d387d\") "
Oct 30 03:25:10.508438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.508403    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume" pod "d4ae5348-7992-40be-9d82-b543821d387d" (UID: "d4ae5348-7992-40be-9d82-b543821d387d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:10.510041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.510019    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz" pod "d4ae5348-7992-40be-9d82-b543821d387d" (UID: "d4ae5348-7992-40be-9d82-b543821d387d"). InnerVolumeSpecName "kube-api-access-46jtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:10.609040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.608938    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/d4ae5348-7992-40be-9d82-b543821d387d-kube-api-access-46jtz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:25:10.609040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.608968    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ae5348-7992-40be-9d82-b543821d387d-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:25:10.824570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.824540    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"]
Oct 30 03:25:10.835630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:10.835603    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0f003-7b46ddc7b4-k8ctd"]
Oct 30 03:25:11.107097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.107061    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"]
Oct 30 03:25:11.107964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.107938    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d4ae5348-7992-40be-9d82-b543821d387d" containerName="payload-processing"
Oct 30 03:25:11.108051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.107967    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ae5348-7992-40be-9d82-b543821d387d" containerName="payload-processing"
Oct 30 03:25:11.108051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.107991    4877 cpu_manager.go:414] "containerMap: removing container" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy"
Oct 30 03:25:11.108051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.107998    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy"
Oct 30 03:25:11.108051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108025    4877 cpu_manager.go:414] "containerMap: removing container" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy"
Oct 30 03:25:11.108051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108033    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy"
Oct 30 03:25:11.108051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108050    4877 cpu_manager.go:414] "containerMap: removing container" podUID="77425bc0-2a1a-445f-8254-54e977a01b41" containerName="payload-pre-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108058    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77425bc0-2a1a-445f-8254-54e977a01b41" containerName="payload-pre-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108071    4877 cpu_manager.go:414] "containerMap: removing container" podUID="de555438-38ef-487f-88e2-dfb7e7459f0a" containerName="payload-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108081    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de555438-38ef-487f-88e2-dfb7e7459f0a" containerName="payload-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108110    4877 cpu_manager.go:414] "containerMap: removing container" podUID="502500f2-9ce1-47e9-88f7-1649164bd142" containerName="payload-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108117    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502500f2-9ce1-47e9-88f7-1649164bd142" containerName="payload-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108129    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108138    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108156    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108163    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108183    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0be95dad-ad7f-4b82-93be-f58082f057db" containerName="payload-pre-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108190    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be95dad-ad7f-4b82-93be-f58082f057db" containerName="payload-pre-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108211    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b16334de-f241-4ebb-8d14-8026394efc82" containerName="payload-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108219    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16334de-f241-4ebb-8d14-8026394efc82" containerName="payload-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108233    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3c7cb16e-37e3-4463-877d-efae398806f0" containerName="payload-pre-processing"
Oct 30 03:25:11.108360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108240    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7cb16e-37e3-4463-877d-efae398806f0" containerName="payload-pre-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108380    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ae5348-7992-40be-9d82-b543821d387d" containerName="payload-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108398    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108408    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="075ae435-693b-406e-ad16-3d253430068b" containerName="istio-proxy"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108418    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="77425bc0-2a1a-445f-8254-54e977a01b41" containerName="payload-pre-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108447    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="de555438-38ef-487f-88e2-dfb7e7459f0a" containerName="payload-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108460    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="502500f2-9ce1-47e9-88f7-1649164bd142" containerName="payload-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108470    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108480    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3b31c2-a4df-4aed-b955-0639253f104c" containerName="istio-proxy"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108491    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be95dad-ad7f-4b82-93be-f58082f057db" containerName="payload-pre-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108503    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16334de-f241-4ebb-8d14-8026394efc82" containerName="payload-processing"
Oct 30 03:25:11.109638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.108513    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7cb16e-37e3-4463-877d-efae398806f0" containerName="payload-pre-processing"
Oct 30 03:25:11.112685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.112664    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.133004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.132973    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"]
Oct 30 03:25:11.139568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.139535    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-91a20d-openshift-default-dockercfg-rjrh6\""
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229099    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229152    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229189    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229218    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229246    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvlb\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229355    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229402    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229457    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.236482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.229590    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.330722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.330687    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.330925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.330783    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.330925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.330826    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.330925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.330842    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.330925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.330864    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.330990    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331027    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvlb\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331128    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331175    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331182    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331238    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331509    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331672    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.331813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.331699    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.333258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.333239    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.333490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.333469    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.350116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.350085    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.350274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.350195    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvlb\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb\") pod \"e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") " pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.429855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.429761    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:11.699871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.699800    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ae5348-7992-40be-9d82-b543821d387d" path="/var/lib/kubelet/pods/d4ae5348-7992-40be-9d82-b543821d387d/volumes"
Oct 30 03:25:11.751362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.751333    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"]
Oct 30 03:25:11.751640 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:11.751610    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa8774c_c189_4cbf_81a1_b3466cc1250f.slice/crio-da26fbc5e58cee6fc88b50538b47f97a06ed9154cd88b4e0d0433529e421a9f5 WatchSource:0}: Error finding container da26fbc5e58cee6fc88b50538b47f97a06ed9154cd88b4e0d0433529e421a9f5: Status 404 returned error can't find the container with id da26fbc5e58cee6fc88b50538b47f97a06ed9154cd88b4e0d0433529e421a9f5
Oct 30 03:25:11.753992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.753963    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:11.754058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.754024    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:11.754058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:11.754053    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:12.576699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.576278    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" event={"ID":"bfa8774c-c189-4cbf-81a1-b3466cc1250f","Type":"ContainerStarted","Data":"b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"}
Oct 30 03:25:12.576699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.576352    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" event={"ID":"bfa8774c-c189-4cbf-81a1-b3466cc1250f","Type":"ContainerStarted","Data":"da26fbc5e58cee6fc88b50538b47f97a06ed9154cd88b4e0d0433529e421a9f5"}
Oct 30 03:25:12.581319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.581295    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"]
Oct 30 03:25:12.587460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.587438    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"]
Oct 30 03:25:12.587757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.587704    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.604972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.604948    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b03a5e-dockercfg-pdn47\""
Oct 30 03:25:12.605104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.605011    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b03a5e\""
Oct 30 03:25:12.640090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.640036    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podStartSLOduration=1.640021237 podStartE2EDuration="1.640021237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:12.63782542 +0000 UTC m=+1341.066714868" watchObservedRunningTime="2026-10-30 03:25:12.640021237 +0000 UTC m=+1341.068910655"
Oct 30 03:25:12.653051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.653016    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xll8\" (UniqueName: \"kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.653254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.653220    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.714250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.714216    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"]
Oct 30 03:25:12.718268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.718250    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:12.746417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.746392    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"]
Oct 30 03:25:12.815609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.813504    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.815609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.813692    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:12.815609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.813790    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xll8\" (UniqueName: \"kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.815609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.813881    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tbj\" (UniqueName: \"kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj\") pod \"payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:12.816400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.816357    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.834025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.833961    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xll8\" (UniqueName: \"kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.915411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.915362    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:12.915646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.915540    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tbj\" (UniqueName: \"kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj\") pod \"payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:12.916159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.916134    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:12.917894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.917870    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:12.935375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:12.935349    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tbj\" (UniqueName: \"kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj\") pod \"payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:13.033321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.033282    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:13.226158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.226131    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"]
Oct 30 03:25:13.226703 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:13.226679    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb29ede0_78e1_428f_80f1_886769fbd3c9.slice/crio-a66fb3c352ee37a7fb4ec24a072397fe582ec233dc7d2d40214114f54f673c1b WatchSource:0}: Error finding container a66fb3c352ee37a7fb4ec24a072397fe582ec233dc7d2d40214114f54f673c1b: Status 404 returned error can't find the container with id a66fb3c352ee37a7fb4ec24a072397fe582ec233dc7d2d40214114f54f673c1b
Oct 30 03:25:13.344236 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:13.344201    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82f245_4355_47fa_b2f7_72f041cae8e4.slice/crio-736ff5bd538ff88fa7e625b6dd5112953884db1bf5a93f9aea4b3fb2408167bd WatchSource:0}: Error finding container 736ff5bd538ff88fa7e625b6dd5112953884db1bf5a93f9aea4b3fb2408167bd: Status 404 returned error can't find the container with id 736ff5bd538ff88fa7e625b6dd5112953884db1bf5a93f9aea4b3fb2408167bd
Oct 30 03:25:13.345709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.345677    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"]
Oct 30 03:25:13.432350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.432302    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:13.440008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.439326    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:13.585120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.585079    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4" event={"ID":"fb29ede0-78e1-428f-80f1-886769fbd3c9","Type":"ContainerStarted","Data":"b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906"}
Oct 30 03:25:13.585549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.585151    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:13.585549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.585175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4" event={"ID":"fb29ede0-78e1-428f-80f1-886769fbd3c9","Type":"ContainerStarted","Data":"a66fb3c352ee37a7fb4ec24a072397fe582ec233dc7d2d40214114f54f673c1b"}
Oct 30 03:25:13.586986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.586953    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6" event={"ID":"7d82f245-4355-47fa-b2f7-72f041cae8e4","Type":"ContainerStarted","Data":"64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440"}
Oct 30 03:25:13.587148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.587011    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6" event={"ID":"7d82f245-4355-47fa-b2f7-72f041cae8e4","Type":"ContainerStarted","Data":"736ff5bd538ff88fa7e625b6dd5112953884db1bf5a93f9aea4b3fb2408167bd"}
Oct 30 03:25:13.587207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.587157    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:13.587253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.587205    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:13.588182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.588163    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:25:13.626005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:13.625848    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4" podStartSLOduration=1.625832724 podStartE2EDuration="1.625832724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:13.623196637 +0000 UTC m=+1342.052086054" watchObservedRunningTime="2026-10-30 03:25:13.625832724 +0000 UTC m=+1342.054722142"
Oct 30 03:25:16.230892 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:25:16.230847    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:25:17.221761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.221699    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6" podStartSLOduration=5.221683293 podStartE2EDuration="5.221683293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:13.719567537 +0000 UTC m=+1342.148456955" watchObservedRunningTime="2026-10-30 03:25:17.221683293 +0000 UTC m=+1345.650572711"
Oct 30 03:25:17.222121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.222103    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"]
Oct 30 03:25:17.234878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.234850    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.248895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.248869    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-91a20d\""
Oct 30 03:25:17.260715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.260683    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-91a20d-dockercfg-4rlpc\""
Oct 30 03:25:17.264961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.264935    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"]
Oct 30 03:25:17.311547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.311510    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"]
Oct 30 03:25:17.317217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.317192    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.342539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.342513    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"]
Oct 30 03:25:17.375997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.375958    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55gxr\" (UniqueName: \"kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr\") pod \"payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.376170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.376011    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.479487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.477279    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55gxr\" (UniqueName: \"kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr\") pod \"payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.479487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.477331    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.479487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.477447    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.479487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.477497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgf4\" (UniqueName: \"kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4\") pod \"payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.479487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.478654    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.501774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.497535    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55gxr\" (UniqueName: \"kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr\") pod \"payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.548621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.548585    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:17.578988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.578949    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.579154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.579022    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgf4\" (UniqueName: \"kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4\") pod \"payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.579962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.579923    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.655027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.654929    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgf4\" (UniqueName: \"kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4\") pod \"payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:17.892132 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:17.892086    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8366e7_8e68_4622_912a_836f734e332f.slice/crio-d83b1e25d64b190723d64c97eac30d38697ce7287e6ab4eabfb477a316abfc51 WatchSource:0}: Error finding container d83b1e25d64b190723d64c97eac30d38697ce7287e6ab4eabfb477a316abfc51: Status 404 returned error can't find the container with id d83b1e25d64b190723d64c97eac30d38697ce7287e6ab4eabfb477a316abfc51
Oct 30 03:25:17.893132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.893102    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"]
Oct 30 03:25:17.933437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:17.933395    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:18.262752 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:18.262721    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf39f530_e107_4708_9ed3_073af8fb8cb6.slice/crio-96af043f4973dc6cbb8070550798d02413708ef9886e32e02c3e6e345eae3a7e WatchSource:0}: Error finding container 96af043f4973dc6cbb8070550798d02413708ef9886e32e02c3e6e345eae3a7e: Status 404 returned error can't find the container with id 96af043f4973dc6cbb8070550798d02413708ef9886e32e02c3e6e345eae3a7e
Oct 30 03:25:18.264204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.264176    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"]
Oct 30 03:25:18.680595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.680554    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn" event={"ID":"4d8366e7-8e68-4622-912a-836f734e332f","Type":"ContainerStarted","Data":"d0f141d88d1e87cb5fc9248ca65c1236e53dc9b874f501f5f6b62e2faa794c65"}
Oct 30 03:25:18.680784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.680630    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn" event={"ID":"4d8366e7-8e68-4622-912a-836f734e332f","Type":"ContainerStarted","Data":"d83b1e25d64b190723d64c97eac30d38697ce7287e6ab4eabfb477a316abfc51"}
Oct 30 03:25:18.680882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.680789    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:18.682379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.682353    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh" event={"ID":"af39f530-e107-4708-9ed3-073af8fb8cb6","Type":"ContainerStarted","Data":"cea9275cc68391754a0e462fb2fb6994e9d3420bd480a6d72f6a354a1778bd85"}
Oct 30 03:25:18.682517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.682400    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh" event={"ID":"af39f530-e107-4708-9ed3-073af8fb8cb6","Type":"ContainerStarted","Data":"96af043f4973dc6cbb8070550798d02413708ef9886e32e02c3e6e345eae3a7e"}
Oct 30 03:25:18.682517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.682443    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:18.721312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.721248    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn" podStartSLOduration=1.721232278 podStartE2EDuration="1.721232278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:18.718057638 +0000 UTC m=+1347.146947055" watchObservedRunningTime="2026-10-30 03:25:18.721232278 +0000 UTC m=+1347.150121696"
Oct 30 03:25:18.763919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:18.763856    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh" podStartSLOduration=1.763840617 podStartE2EDuration="1.763840617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:18.760337959 +0000 UTC m=+1347.189227378" watchObservedRunningTime="2026-10-30 03:25:18.763840617 +0000 UTC m=+1347.192730036"
Oct 30 03:25:20.471507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.471474    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"]
Oct 30 03:25:20.476361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.476338    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.508666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.508635    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"]
Oct 30 03:25:20.541239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541199    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541241    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541269    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541369    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541409    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541531    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4q7\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541570    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541636    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.541771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.541676    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.555567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.555540    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7ce670-openshift-default-dockercfg-qq76l\""
Oct 30 03:25:20.642545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642506    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4q7\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.642743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642560    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.642743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642596    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.642743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642617    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.642743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642657    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.642743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642689    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.642743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642730    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642787    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.642890    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.643192    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.643244    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.643272    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.643330    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.643618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.643596    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.645064 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.645039    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.645344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.645324    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.661048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.661022    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.661338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.661313    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4q7\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7\") pod \"e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") " pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:20.811298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:20.811248    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:21.144540 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:21.144509    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530e5791_dc0a_4d50_80c7_61241a65b260.slice/crio-9c6142ddc2589d2b6bdbd11bb5a12180a590ea70b9cf0140ecf8784becff408c WatchSource:0}: Error finding container 9c6142ddc2589d2b6bdbd11bb5a12180a590ea70b9cf0140ecf8784becff408c: Status 404 returned error can't find the container with id 9c6142ddc2589d2b6bdbd11bb5a12180a590ea70b9cf0140ecf8784becff408c
Oct 30 03:25:21.144832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.144809    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"]
Oct 30 03:25:21.147078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.147047    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:21.147138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.147128    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:21.147170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.147159    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:21.770450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.762398    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" event={"ID":"530e5791-dc0a-4d50-80c7-61241a65b260","Type":"ContainerStarted","Data":"25b2a0290070718a91fd09016dbbe9193786065b0dd82a5f796f992a577b04f3"}
Oct 30 03:25:21.770450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.762496    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" event={"ID":"530e5791-dc0a-4d50-80c7-61241a65b260","Type":"ContainerStarted","Data":"9c6142ddc2589d2b6bdbd11bb5a12180a590ea70b9cf0140ecf8784becff408c"}
Oct 30 03:25:21.814523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.812015    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:21.834497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:21.834440    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podStartSLOduration=1.834409209 podStartE2EDuration="1.834409209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:21.830245973 +0000 UTC m=+1350.259135393" watchObservedRunningTime="2026-10-30 03:25:21.834409209 +0000 UTC m=+1350.263298627"
Oct 30 03:25:22.817918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:22.817887    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:23.777439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:23.777399    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:23.778336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:23.778312    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:25:24.596981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:24.596949    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:25:24.597497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:24.597045    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:25:28.893481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.893344    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"]
Oct 30 03:25:28.899102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.899077    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.913330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.913304    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 03:25:28.913520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.913498    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 03:25:28.924905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.924875    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 03:25:28.925109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.925087    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ttlms\""
Oct 30 03:25:28.931278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.931249    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"]
Oct 30 03:25:28.956479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.956417    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b98\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.956654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.956581    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.956714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.956658    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.956758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.956746    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.956815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.956770    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.956911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.956889    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:28.957036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:28.957018    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058191    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058273    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b98\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058323    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058348    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058396    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058419    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.058495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.058482    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.059374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.059345    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.060834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.060809    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.060935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.060823    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.060982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.060949    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.061118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.061096    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.076963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.076935    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.077385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.077362    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b98\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9p5qn\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.213264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.213186    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6_7d82f245-4355-47fa-b2f7-72f041cae8e4/payload-processing/0.log"
Oct 30 03:25:29.216884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.216863    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:29.538749 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:25:29.538711    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6556ea60_dcf4_4020_bdb1_15fb07e71c82.slice/crio-f7f5baec2244f7670e9d72add1ff6fb2220b85b6566c2537062f6b27423944e2 WatchSource:0}: Error finding container f7f5baec2244f7670e9d72add1ff6fb2220b85b6566c2537062f6b27423944e2: Status 404 returned error can't find the container with id f7f5baec2244f7670e9d72add1ff6fb2220b85b6566c2537062f6b27423944e2
Oct 30 03:25:29.539323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.539299    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"]
Oct 30 03:25:29.701185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.701157    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:25:29.701367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.701219    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:25:29.825504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:29.825388    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn" event={"ID":"6556ea60-dcf4-4020-bdb1-15fb07e71c82","Type":"ContainerStarted","Data":"f7f5baec2244f7670e9d72add1ff6fb2220b85b6566c2537062f6b27423944e2"}
Oct 30 03:25:32.461866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:32.461830    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:32.462195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:32.461897    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:25:32.853618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:32.853577    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn" event={"ID":"6556ea60-dcf4-4020-bdb1-15fb07e71c82","Type":"ContainerStarted","Data":"b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5"}
Oct 30 03:25:32.853804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:32.853672    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:32.924027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:32.923965    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn" podStartSLOduration=2.004098852 podStartE2EDuration="4.92394802s" totalImagesPullingTime="2.919849168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:32.910978407 +0000 UTC m=+1361.339867836" watchObservedRunningTime="2026-10-30 03:25:32.92394802 +0000 UTC m=+1361.352837441"
Oct 30 03:25:33.044176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:33.044110    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"]
Oct 30 03:25:33.049940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:33.049900    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf9dc14-6vhjg"]
Oct 30 03:25:33.701692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:33.701655    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a0c923-ce8e-4f30-b88c-e828c0a2ba30" path="/var/lib/kubelet/pods/e5a0c923-ce8e-4f30-b88c-e828c0a2ba30/volumes"
Oct 30 03:25:33.864303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:25:33.864275    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:25:46.231379 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:25:46.231342    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:26:01.601661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:01.601623    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"]
Oct 30 03:26:01.602220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:01.601905    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779" podUID="54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" containerName="payload-processing" containerID="cri-o://a83ab68734843a90613a5bf6626c8884cbe0048d2796fdb24ca20544235ca63f" gracePeriod=30
Oct 30 03:26:01.629788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:01.626652    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"]
Oct 30 03:26:01.629788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:01.626917    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs" podUID="9fc2c451-934c-4ca0-b73a-9717f242250a" containerName="payload-pre-processing" containerID="cri-o://da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f" gracePeriod=30
Oct 30 03:26:02.120619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.120587    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:26:02.126082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.125659    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume\") pod \"9fc2c451-934c-4ca0-b73a-9717f242250a\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") "
Oct 30 03:26:02.126082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.125926    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs\" (UniqueName: \"kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs\") pod \"9fc2c451-934c-4ca0-b73a-9717f242250a\" (UID: \"9fc2c451-934c-4ca0-b73a-9717f242250a\") "
Oct 30 03:26:02.126082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.126056    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume" pod "9fc2c451-934c-4ca0-b73a-9717f242250a" (UID: "9fc2c451-934c-4ca0-b73a-9717f242250a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:02.127102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.127070    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc2c451-934c-4ca0-b73a-9717f242250a-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:02.129466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.128711    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs" pod "9fc2c451-934c-4ca0-b73a-9717f242250a" (UID: "9fc2c451-934c-4ca0-b73a-9717f242250a"). InnerVolumeSpecName "kube-api-access-sqfcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:02.144550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.144522    4877 generic.go:356] "Generic (PLEG): container finished" podID="54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" containerID="a83ab68734843a90613a5bf6626c8884cbe0048d2796fdb24ca20544235ca63f" exitCode=0
Oct 30 03:26:02.144729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.144707    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779" event={"ID":"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc","Type":"ContainerDied","Data":"a83ab68734843a90613a5bf6626c8884cbe0048d2796fdb24ca20544235ca63f"}
Oct 30 03:26:02.146791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.146764    4877 generic.go:356] "Generic (PLEG): container finished" podID="9fc2c451-934c-4ca0-b73a-9717f242250a" containerID="da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f" exitCode=0
Oct 30 03:26:02.146893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.146836    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"
Oct 30 03:26:02.146983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.146966    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs" event={"ID":"9fc2c451-934c-4ca0-b73a-9717f242250a","Type":"ContainerDied","Data":"da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f"}
Oct 30 03:26:02.147046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.147004    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs" event={"ID":"9fc2c451-934c-4ca0-b73a-9717f242250a","Type":"ContainerDied","Data":"8aa2da98eb13e0ff2012a4babed01e5712765129ebb2b729ae1017ccdf5e781b"}
Oct 30 03:26:02.147046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.147030    4877 scope.go:122] "RemoveContainer" containerID="da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f"
Oct 30 03:26:02.165216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.165193    4877 scope.go:122] "RemoveContainer" containerID="da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f"
Oct 30 03:26:02.165803 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:26:02.165769    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f\": container with ID starting with da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f not found: ID does not exist" containerID="da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f"
Oct 30 03:26:02.165908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.165815    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f"} err="failed to get container status \"da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f\": rpc error: code = NotFound desc = could not find container \"da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f\": container with ID starting with da1384dc304201d4a94d29d34e2eb5ecaa0e1b900b066ba79228bb6553d4759f not found: ID does not exist"
Oct 30 03:26:02.216097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.216060    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"]
Oct 30 03:26:02.220758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.220725    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-499bca-57d6795ff9-mftcs"]
Oct 30 03:26:02.225914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.225894    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:26:02.228209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.228177    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume\") pod \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") "
Oct 30 03:26:02.228379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.228367    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt\" (UniqueName: \"kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt\") pod \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\" (UID: \"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc\") "
Oct 30 03:26:02.228773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.228751    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume" pod "54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" (UID: "54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:02.229025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.229005    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfcs\" (UniqueName: \"kubernetes.io/projected/9fc2c451-934c-4ca0-b73a-9717f242250a-kube-api-access-sqfcs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:02.229143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.229127    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:02.231438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.231405    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt" pod "54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" (UID: "54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc"). InnerVolumeSpecName "kube-api-access-ggtvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:02.330303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:02.330210    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtvt\" (UniqueName: \"kubernetes.io/projected/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc-kube-api-access-ggtvt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:03.156687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.156652    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"
Oct 30 03:26:03.157199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.156689    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779" event={"ID":"54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc","Type":"ContainerDied","Data":"ef2a561ed123e99905a93df0989c250ee7d1d9ba77b8648a9ed722f5cf5a957c"}
Oct 30 03:26:03.157199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.156756    4877 scope.go:122] "RemoveContainer" containerID="a83ab68734843a90613a5bf6626c8884cbe0048d2796fdb24ca20544235ca63f"
Oct 30 03:26:03.214815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.214777    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"]
Oct 30 03:26:03.225661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.225628    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-499bca-67c4f5d754-x4779"]
Oct 30 03:26:03.741508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.740278    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" path="/var/lib/kubelet/pods/54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc/volumes"
Oct 30 03:26:03.750449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:03.745521    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc2c451-934c-4ca0-b73a-9717f242250a" path="/var/lib/kubelet/pods/9fc2c451-934c-4ca0-b73a-9717f242250a/volumes"
Oct 30 03:26:09.828095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:09.827597    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:26:09.828095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:09.827686    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:26:11.048349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:11.048301    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"]
Oct 30 03:26:11.049574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:11.049532    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy" containerID="cri-o://7612037a50cbb104dc0294a08c9eb1ffc39319686710b4f6f2ba642b416f94ff" gracePeriod=30
Oct 30 03:26:13.284678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.284634    4877 generic.go:356] "Generic (PLEG): container finished" podID="530e5791-dc0a-4d50-80c7-61241a65b260" containerID="25b2a0290070718a91fd09016dbbe9193786065b0dd82a5f796f992a577b04f3" exitCode=0
Oct 30 03:26:13.287187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.284710    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" event={"ID":"530e5791-dc0a-4d50-80c7-61241a65b260","Type":"ContainerDied","Data":"25b2a0290070718a91fd09016dbbe9193786065b0dd82a5f796f992a577b04f3"}
Oct 30 03:26:13.287187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.285219    4877 scope.go:122] "RemoveContainer" containerID="25b2a0290070718a91fd09016dbbe9193786065b0dd82a5f796f992a577b04f3"
Oct 30 03:26:13.287187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.286936    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:13.287187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.287004    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:13.287187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.287035    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:13.596508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.588076    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:13.596508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:13.588133    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:14.314274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.314234    4877 generic.go:356] "Generic (PLEG): container finished" podID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerID="b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3" exitCode=0
Oct 30 03:26:14.314727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.314275    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" event={"ID":"bfa8774c-c189-4cbf-81a1-b3466cc1250f","Type":"ContainerDied","Data":"b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"}
Oct 30 03:26:14.314783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.314725    4877 scope.go:122] "RemoveContainer" containerID="b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"
Oct 30 03:26:14.316489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.316418    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:14.316608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.316547    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:14.316608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.316598    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:14.316896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.316876    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" event={"ID":"530e5791-dc0a-4d50-80c7-61241a65b260","Type":"ContainerStarted","Data":"80d8da0e70675c273c7f07ef6afcfa830c80dceca4aae24c0799bb0aac14f78c"}
Oct 30 03:26:14.812480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.812439    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:26:14.812749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.812723    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:26:14.812846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:14.812785    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 03:26:15.340225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:15.340190    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" event={"ID":"bfa8774c-c189-4cbf-81a1-b3466cc1250f","Type":"ContainerStarted","Data":"fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5"}
Oct 30 03:26:15.433294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:15.433246    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:26:15.434760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:15.434599    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:15.434983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:15.434959    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:15.832455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:15.831967    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:26:15.832455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:15.832046    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 03:26:16.294448 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:26:16.289938    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:26:16.364448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.360207    4877 generic.go:356] "Generic (PLEG): container finished" podID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerID="7612037a50cbb104dc0294a08c9eb1ffc39319686710b4f6f2ba642b416f94ff" exitCode=0
Oct 30 03:26:16.364448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.360276    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" event={"ID":"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e","Type":"ContainerDied","Data":"7612037a50cbb104dc0294a08c9eb1ffc39319686710b4f6f2ba642b416f94ff"}
Oct 30 03:26:16.364448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.360369    4877 scope.go:122] "RemoveContainer" containerID="1ea355187b043e4f78a8542a467dd515d46cf62d00c4e91c94495082388dbf7c"
Oct 30 03:26:16.458659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.456889    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:16.458659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.456957    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:16.553893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.553865    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:26:16.571305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571278    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571322    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571393    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571495    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571542    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571567    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:16.571632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571576    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571684    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571726    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571759    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket\") pod \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\" (UID: \"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e\") "
Oct 30 03:26:16.571848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571807    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:16.571848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.571796    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:16.572163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.572021    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:16.572240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.572212    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:16.572328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.572306    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.572372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.572335    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.572372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.572351    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.572372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.572363    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.575009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.574968    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:26:16.575707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.575668    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "kube-api-access-5qtm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:16.575920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.575890    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:26:16.576058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.576032    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token" pod "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" (UID: "f2c0f2b4-47f1-4138-a91b-9c2aa700d32e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:16.673900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.673822    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.673900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.673850    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtm7\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-kube-api-access-5qtm7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.673900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.673861    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.673900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.673870    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.673900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.673878    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:16.819672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.819633    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:26:16.819971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:16.819938    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 03:26:17.374464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.370896    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll" event={"ID":"f2c0f2b4-47f1-4138-a91b-9c2aa700d32e","Type":"ContainerDied","Data":"51768b5ec51a4f54144f9593c9ac4105d3ff2c52067300272387842f80b80870"}
Oct 30 03:26:17.374464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.370930    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"
Oct 30 03:26:17.374464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.370954    4877 scope.go:122] "RemoveContainer" containerID="7612037a50cbb104dc0294a08c9eb1ffc39319686710b4f6f2ba642b416f94ff"
Oct 30 03:26:17.437935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.437902    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:17.438226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.438196    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:17.460290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.460257    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"]
Oct 30 03:26:17.471559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.471531    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-499bca-openshift-default-769dd4644c-7vvll"]
Oct 30 03:26:17.741604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.741456    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" path="/var/lib/kubelet/pods/f2c0f2b4-47f1-4138-a91b-9c2aa700d32e/volumes"
Oct 30 03:26:17.812024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.811986    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:26:17.812344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:17.812298    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 03:26:18.439304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:18.439265    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:18.439841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:18.439326    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:18.811873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:18.811828    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 03:26:18.812165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:18.812134    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 03:26:19.440624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:19.440191    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:19.440624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:19.440248    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:19.964325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:19.964288    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:26:19.964986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:19.964963    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:26:19.966403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:19.966379    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:26:20.430306 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:20.430271    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 03:26:20.430516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:20.430322    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 03:26:21.435151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:21.435118    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:26:21.435607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:21.435500    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:26:21.436219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:21.436196    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:26:22.074342 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.074302    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"]
Oct 30 03:26:22.075298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075275    4877 cpu_manager.go:414] "containerMap: removing container" podUID="54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" containerName="payload-processing"
Oct 30 03:26:22.075354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075302    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" containerName="payload-processing"
Oct 30 03:26:22.075389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075364    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9fc2c451-934c-4ca0-b73a-9717f242250a" containerName="payload-pre-processing"
Oct 30 03:26:22.075389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075372    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc2c451-934c-4ca0-b73a-9717f242250a" containerName="payload-pre-processing"
Oct 30 03:26:22.075498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075394    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy"
Oct 30 03:26:22.075498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075401    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy"
Oct 30 03:26:22.075498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075416    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy"
Oct 30 03:26:22.075498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075438    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy"
Oct 30 03:26:22.075622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075581    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc2c451-934c-4ca0-b73a-9717f242250a" containerName="payload-pre-processing"
Oct 30 03:26:22.075622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075599    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy"
Oct 30 03:26:22.075622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075610    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c0f2b4-47f1-4138-a91b-9c2aa700d32e" containerName="istio-proxy"
Oct 30 03:26:22.075716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.075625    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="54dcfaa7-1a91-4f6b-8d9c-0705b1b2dcbc" containerName="payload-processing"
Oct 30 03:26:22.080959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.080933    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.104922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.104889    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"]
Oct 30 03:26:22.107488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.107386    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ad5839-openshift-default-dockercfg-dmt8b\""
Oct 30 03:26:22.188920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.188858    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.188931    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189047    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189103    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4fd\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189198    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189251    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189386    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189459    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.189758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.189737    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.290795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.290744    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.290829    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.290869    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.290900    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.290936    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4fd\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.291028    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.291078    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.291198    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.291253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.291236    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.292522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.292478    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.293069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.292771    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.293596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.293570    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.293915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.293891    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.294417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.294395    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.298408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.298331    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.299145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.299119    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.312806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.312777    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.313460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.313418    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4fd\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd\") pod \"e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") " pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.401855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.401774    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:22.725018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.724988    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"]
Oct 30 03:26:22.726772 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:26:22.726745    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a95bd50_498b_4499_bd1e_e0b22b72de18.slice/crio-c1637cfea135887f411fa364ee5acb3b91c7484ff77165447b936c50134a20df WatchSource:0}: Error finding container c1637cfea135887f411fa364ee5acb3b91c7484ff77165447b936c50134a20df: Status 404 returned error can't find the container with id c1637cfea135887f411fa364ee5acb3b91c7484ff77165447b936c50134a20df
Oct 30 03:26:22.729640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.729605    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:22.729727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.729673    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:22.729727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:22.729705    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:23.476269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:23.476233    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" event={"ID":"8a95bd50-498b-4499-bd1e-e0b22b72de18","Type":"ContainerStarted","Data":"bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"}
Oct 30 03:26:23.476481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:23.476300    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" event={"ID":"8a95bd50-498b-4499-bd1e-e0b22b72de18","Type":"ContainerStarted","Data":"c1637cfea135887f411fa364ee5acb3b91c7484ff77165447b936c50134a20df"}
Oct 30 03:26:23.533338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:23.533287    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podStartSLOduration=1.533271974 podStartE2EDuration="1.533271974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:23.530033734 +0000 UTC m=+1411.958923172" watchObservedRunningTime="2026-10-30 03:26:23.533271974 +0000 UTC m=+1411.962161392"
Oct 30 03:26:24.410938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:24.410907    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:24.426482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:24.426415    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:24.484594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:24.484559    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:24.485547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:24.485520    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:26:25.518921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.518884    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"]
Oct 30 03:26:25.519389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.519098    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6" podUID="7d82f245-4355-47fa-b2f7-72f041cae8e4" containerName="payload-processing" containerID="cri-o://64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440" gracePeriod=30
Oct 30 03:26:25.551797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.551760    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"]
Oct 30 03:26:25.552226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.552194    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4" podUID="fb29ede0-78e1-428f-80f1-886769fbd3c9" containerName="payload-pre-processing" containerID="cri-o://b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906" gracePeriod=30
Oct 30 03:26:25.859669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.859634    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:26:25.945021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.943811    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume\") pod \"7d82f245-4355-47fa-b2f7-72f041cae8e4\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") "
Oct 30 03:26:25.945021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.944024    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj\" (UniqueName: \"kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj\") pod \"7d82f245-4355-47fa-b2f7-72f041cae8e4\" (UID: \"7d82f245-4355-47fa-b2f7-72f041cae8e4\") "
Oct 30 03:26:25.945890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.945821    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume" pod "7d82f245-4355-47fa-b2f7-72f041cae8e4" (UID: "7d82f245-4355-47fa-b2f7-72f041cae8e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:25.973850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.973787    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj" pod "7d82f245-4355-47fa-b2f7-72f041cae8e4" (UID: "7d82f245-4355-47fa-b2f7-72f041cae8e4"). InnerVolumeSpecName "kube-api-access-47tbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:25.988492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:25.986852    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:26:26.045465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.045410    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47tbj\" (UniqueName: \"kubernetes.io/projected/7d82f245-4355-47fa-b2f7-72f041cae8e4-kube-api-access-47tbj\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:26.045465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.045456    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d82f245-4355-47fa-b2f7-72f041cae8e4-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:26.146006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.145911    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume\") pod \"fb29ede0-78e1-428f-80f1-886769fbd3c9\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") "
Oct 30 03:26:26.146006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.145965    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8\" (UniqueName: \"kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8\") pod \"fb29ede0-78e1-428f-80f1-886769fbd3c9\" (UID: \"fb29ede0-78e1-428f-80f1-886769fbd3c9\") "
Oct 30 03:26:26.146386 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.146359    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume" pod "fb29ede0-78e1-428f-80f1-886769fbd3c9" (UID: "fb29ede0-78e1-428f-80f1-886769fbd3c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:26:26.148034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.148014    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8" pod "fb29ede0-78e1-428f-80f1-886769fbd3c9" (UID: "fb29ede0-78e1-428f-80f1-886769fbd3c9"). InnerVolumeSpecName "kube-api-access-5xll8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:26:26.247664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.247630    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb29ede0-78e1-428f-80f1-886769fbd3c9-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:26.247664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.247659    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xll8\" (UniqueName: \"kubernetes.io/projected/fb29ede0-78e1-428f-80f1-886769fbd3c9-kube-api-access-5xll8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:26:26.502716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.502622    4877 generic.go:356] "Generic (PLEG): container finished" podID="fb29ede0-78e1-428f-80f1-886769fbd3c9" containerID="b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906" exitCode=0
Oct 30 03:26:26.502716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.502669    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"
Oct 30 03:26:26.502939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.502709    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4" event={"ID":"fb29ede0-78e1-428f-80f1-886769fbd3c9","Type":"ContainerDied","Data":"b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906"}
Oct 30 03:26:26.502939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.502781    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4" event={"ID":"fb29ede0-78e1-428f-80f1-886769fbd3c9","Type":"ContainerDied","Data":"a66fb3c352ee37a7fb4ec24a072397fe582ec233dc7d2d40214114f54f673c1b"}
Oct 30 03:26:26.502939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.502813    4877 scope.go:122] "RemoveContainer" containerID="b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906"
Oct 30 03:26:26.504500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.504475    4877 generic.go:356] "Generic (PLEG): container finished" podID="7d82f245-4355-47fa-b2f7-72f041cae8e4" containerID="64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440" exitCode=0
Oct 30 03:26:26.504645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.504520    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"
Oct 30 03:26:26.504645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.504567    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6" event={"ID":"7d82f245-4355-47fa-b2f7-72f041cae8e4","Type":"ContainerDied","Data":"64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440"}
Oct 30 03:26:26.504645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.504612    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6" event={"ID":"7d82f245-4355-47fa-b2f7-72f041cae8e4","Type":"ContainerDied","Data":"736ff5bd538ff88fa7e625b6dd5112953884db1bf5a93f9aea4b3fb2408167bd"}
Oct 30 03:26:26.515498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.515479    4877 scope.go:122] "RemoveContainer" containerID="b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906"
Oct 30 03:26:26.515777 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:26:26.515759    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906\": container with ID starting with b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906 not found: ID does not exist" containerID="b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906"
Oct 30 03:26:26.515838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.515784    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906"} err="failed to get container status \"b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906\": rpc error: code = NotFound desc = could not find container \"b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906\": container with ID starting with b26731d35b81521735f1e55499d1a3ce4d466db7a00cea3b3a0e6c3ce4803906 not found: ID does not exist"
Oct 30 03:26:26.515838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.515802    4877 scope.go:122] "RemoveContainer" containerID="64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440"
Oct 30 03:26:26.525998 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.525845    4877 scope.go:122] "RemoveContainer" containerID="64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440"
Oct 30 03:26:26.526270 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:26:26.526144    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440\": container with ID starting with 64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440 not found: ID does not exist" containerID="64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440"
Oct 30 03:26:26.526270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.526169    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440"} err="failed to get container status \"64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440\": rpc error: code = NotFound desc = could not find container \"64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440\": container with ID starting with 64f5b91b5071bc93d9d2eea5fe55695a191b21d8274ae009a70af2767628a440 not found: ID does not exist"
Oct 30 03:26:26.554548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.554517    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"]
Oct 30 03:26:26.566012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.565982    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b03a5e-6b6fcd7b5b-4t5s6"]
Oct 30 03:26:26.598217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.598189    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"]
Oct 30 03:26:26.608892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:26.608852    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-6c764b9767-xxvb4"]
Oct 30 03:26:27.709396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:27.709359    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d82f245-4355-47fa-b2f7-72f041cae8e4" path="/var/lib/kubelet/pods/7d82f245-4355-47fa-b2f7-72f041cae8e4/volumes"
Oct 30 03:26:27.709909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:27.709892    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb29ede0-78e1-428f-80f1-886769fbd3c9" path="/var/lib/kubelet/pods/fb29ede0-78e1-428f-80f1-886769fbd3c9/volumes"
Oct 30 03:26:29.029561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:29.029523    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"]
Oct 30 03:26:29.041267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:29.041231    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3873930d-gjp5j"]
Oct 30 03:26:29.701080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:29.701046    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbd4589-78f2-4987-8955-4c20fab7c060" path="/var/lib/kubelet/pods/9bbd4589-78f2-4987-8955-4c20fab7c060/volumes"
Oct 30 03:26:36.761142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.761105    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"]
Oct 30 03:26:36.761928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.761905    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fb29ede0-78e1-428f-80f1-886769fbd3c9" containerName="payload-pre-processing"
Oct 30 03:26:36.761928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.761926    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb29ede0-78e1-428f-80f1-886769fbd3c9" containerName="payload-pre-processing"
Oct 30 03:26:36.762189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.761946    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7d82f245-4355-47fa-b2f7-72f041cae8e4" containerName="payload-processing"
Oct 30 03:26:36.762189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.761953    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d82f245-4355-47fa-b2f7-72f041cae8e4" containerName="payload-processing"
Oct 30 03:26:36.762189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.762157    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb29ede0-78e1-428f-80f1-886769fbd3c9" containerName="payload-pre-processing"
Oct 30 03:26:36.762189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.762180    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d82f245-4355-47fa-b2f7-72f041cae8e4" containerName="payload-processing"
Oct 30 03:26:36.767361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.767341    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.796652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.787867    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-194506-openshift-default-dockercfg-p72tz\""
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.797087    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.797679    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.797723    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.797824    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.797872    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.797995    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.798024    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.798059    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnkg\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.800294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.798198    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.807785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.807717    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"]
Oct 30 03:26:36.912447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912166    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912289    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912339    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912375    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912500    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912529    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912638    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.912881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.912672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnkg\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.913676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.913500    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.957924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.913762    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.957924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.915044    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.957924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.915284    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.957924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.921957    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.957924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.929033    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.957924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.952440    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnkg\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.971083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.971050    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:36.979845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:36.979811    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy\") pod \"e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") " pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:37.109663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.109620    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:37.445313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.445276    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"]
Oct 30 03:26:37.445625 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:26:37.445597    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9cf411_4e26_4c4c_9716_c7df36175bc2.slice/crio-2bc870309cc82ef42429c156188ce6ed865ad5c56336feb7737f8c5b9cf92b63 WatchSource:0}: Error finding container 2bc870309cc82ef42429c156188ce6ed865ad5c56336feb7737f8c5b9cf92b63: Status 404 returned error can't find the container with id 2bc870309cc82ef42429c156188ce6ed865ad5c56336feb7737f8c5b9cf92b63
Oct 30 03:26:37.448440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.448391    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:37.448561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.448484    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:37.448561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.448529    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:26:37.601070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.599984    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" event={"ID":"0d9cf411-4e26-4c4c-9716-c7df36175bc2","Type":"ContainerStarted","Data":"a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"}
Oct 30 03:26:37.601070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.600050    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" event={"ID":"0d9cf411-4e26-4c4c-9716-c7df36175bc2","Type":"ContainerStarted","Data":"2bc870309cc82ef42429c156188ce6ed865ad5c56336feb7737f8c5b9cf92b63"}
Oct 30 03:26:37.642663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:37.642601    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podStartSLOduration=1.642580755 podStartE2EDuration="1.642580755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:37.641225581 +0000 UTC m=+1426.070115010" watchObservedRunningTime="2026-10-30 03:26:37.642580755 +0000 UTC m=+1426.071470176"
Oct 30 03:26:38.110446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:38.110399    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:38.116647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:38.116620    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:38.607564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:38.607531    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:38.608516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:38.608493    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:26:46.231099 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:26:46.231060    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:26:54.404228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:26:54.404195    4877 scope.go:122] "RemoveContainer" containerID="42e97b4775dc74621ef72f643b54988d7f6b2ea4bd096d37617a37179d20be1b"
Oct 30 03:27:09.486559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.486525    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:27:09.486996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.486583    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 03:27:09.896770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.896731    4877 generic.go:356] "Generic (PLEG): container finished" podID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerID="bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73" exitCode=0
Oct 30 03:27:09.896986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.896775    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" event={"ID":"8a95bd50-498b-4499-bd1e-e0b22b72de18","Type":"ContainerDied","Data":"bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"}
Oct 30 03:27:09.897264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.897245    4877 scope.go:122] "RemoveContainer" containerID="bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"
Oct 30 03:27:09.899105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.899063    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:09.899201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.899163    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:09.899240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:09.899209    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:10.911164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:10.911125    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" event={"ID":"8a95bd50-498b-4499-bd1e-e0b22b72de18","Type":"ContainerStarted","Data":"9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1"}
Oct 30 03:27:11.412772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:11.412733    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:27:11.412945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:11.412917    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:27:11.412998 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:11.412967    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 03:27:12.402495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:12.402411    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:27:12.403026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:12.402519    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 03:27:12.708718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:12.708630    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"]
Oct 30 03:27:12.717371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:12.717336    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"]
Oct 30 03:27:13.411927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:13.411887    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:27:13.412555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:13.412520    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 03:27:14.402911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:14.402862    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:27:14.403107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:14.402935    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 03:27:15.403844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:15.403804    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 03:27:15.404472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:15.404413    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 03:27:16.229851 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:27:16.229813    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:27:16.407464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:16.407400    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:27:16.408105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:16.408083    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:27:16.408731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:16.408714    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:27:17.710609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:17.710572    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh" podUID="af39f530-e107-4708-9ed3-073af8fb8cb6" containerName="payload-processing" containerID="cri-o://cea9275cc68391754a0e462fb2fb6994e9d3420bd480a6d72f6a354a1778bd85" gracePeriod=25
Oct 30 03:27:17.718846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:17.718815    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn" podUID="4d8366e7-8e68-4622-912a-836f734e332f" containerName="payload-pre-processing" containerID="cri-o://d0f141d88d1e87cb5fc9248ca65c1236e53dc9b874f501f5f6b62e2faa794c65" gracePeriod=25
Oct 30 03:27:17.973582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:17.973455    4877 generic.go:356] "Generic (PLEG): container finished" podID="af39f530-e107-4708-9ed3-073af8fb8cb6" containerID="cea9275cc68391754a0e462fb2fb6994e9d3420bd480a6d72f6a354a1778bd85" exitCode=0
Oct 30 03:27:17.973582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:17.973564    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh" event={"ID":"af39f530-e107-4708-9ed3-073af8fb8cb6","Type":"ContainerDied","Data":"cea9275cc68391754a0e462fb2fb6994e9d3420bd480a6d72f6a354a1778bd85"}
Oct 30 03:27:17.975608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:17.975584    4877 generic.go:356] "Generic (PLEG): container finished" podID="4d8366e7-8e68-4622-912a-836f734e332f" containerID="d0f141d88d1e87cb5fc9248ca65c1236e53dc9b874f501f5f6b62e2faa794c65" exitCode=0
Oct 30 03:27:17.976213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:17.976195    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn" event={"ID":"4d8366e7-8e68-4622-912a-836f734e332f","Type":"ContainerDied","Data":"d0f141d88d1e87cb5fc9248ca65c1236e53dc9b874f501f5f6b62e2faa794c65"}
Oct 30 03:27:18.040931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.040896    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:27:18.077335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.077312    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:27:18.172000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.171969    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4\" (UniqueName: \"kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4\") pod \"af39f530-e107-4708-9ed3-073af8fb8cb6\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") "
Oct 30 03:27:18.172161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.172077    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr\" (UniqueName: \"kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr\") pod \"4d8366e7-8e68-4622-912a-836f734e332f\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") "
Oct 30 03:27:18.172208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.172194    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume\") pod \"4d8366e7-8e68-4622-912a-836f734e332f\" (UID: \"4d8366e7-8e68-4622-912a-836f734e332f\") "
Oct 30 03:27:18.172257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.172246    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume\") pod \"af39f530-e107-4708-9ed3-073af8fb8cb6\" (UID: \"af39f530-e107-4708-9ed3-073af8fb8cb6\") "
Oct 30 03:27:18.172608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.172579    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume" pod "4d8366e7-8e68-4622-912a-836f734e332f" (UID: "4d8366e7-8e68-4622-912a-836f734e332f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:18.172730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.172691    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume" pod "af39f530-e107-4708-9ed3-073af8fb8cb6" (UID: "af39f530-e107-4708-9ed3-073af8fb8cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:18.174128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.174110    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4" pod "af39f530-e107-4708-9ed3-073af8fb8cb6" (UID: "af39f530-e107-4708-9ed3-073af8fb8cb6"). InnerVolumeSpecName "kube-api-access-tvgf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:18.174195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.174143    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr" pod "4d8366e7-8e68-4622-912a-836f734e332f" (UID: "4d8366e7-8e68-4622-912a-836f734e332f"). InnerVolumeSpecName "kube-api-access-55gxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:18.273514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.273477    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8366e7-8e68-4622-912a-836f734e332f-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.273514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.273512    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af39f530-e107-4708-9ed3-073af8fb8cb6-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.273707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.273524    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvgf4\" (UniqueName: \"kubernetes.io/projected/af39f530-e107-4708-9ed3-073af8fb8cb6-kube-api-access-tvgf4\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.273707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.273535    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55gxr\" (UniqueName: \"kubernetes.io/projected/4d8366e7-8e68-4622-912a-836f734e332f-kube-api-access-55gxr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:18.984615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.984516    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"
Oct 30 03:27:18.984615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.984536    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh" event={"ID":"af39f530-e107-4708-9ed3-073af8fb8cb6","Type":"ContainerDied","Data":"96af043f4973dc6cbb8070550798d02413708ef9886e32e02c3e6e345eae3a7e"}
Oct 30 03:27:18.984615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.984600    4877 scope.go:122] "RemoveContainer" containerID="cea9275cc68391754a0e462fb2fb6994e9d3420bd480a6d72f6a354a1778bd85"
Oct 30 03:27:18.986603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.986572    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn" event={"ID":"4d8366e7-8e68-4622-912a-836f734e332f","Type":"ContainerDied","Data":"d83b1e25d64b190723d64c97eac30d38697ce7287e6ab4eabfb477a316abfc51"}
Oct 30 03:27:18.986754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.986634    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"
Oct 30 03:27:18.999270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:18.999180    4877 scope.go:122] "RemoveContainer" containerID="d0f141d88d1e87cb5fc9248ca65c1236e53dc9b874f501f5f6b62e2faa794c65"
Oct 30 03:27:19.059908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.059879    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"]
Oct 30 03:27:19.075538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.075511    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-91a20d-84dc8c9dbb-ngshh"]
Oct 30 03:27:19.107074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.107025    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"]
Oct 30 03:27:19.112976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.112947    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-91a20d-764f5d9b79-hk8pn"]
Oct 30 03:27:19.699922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.699884    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8366e7-8e68-4622-912a-836f734e332f" path="/var/lib/kubelet/pods/4d8366e7-8e68-4622-912a-836f734e332f/volumes"
Oct 30 03:27:19.700397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.700375    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af39f530-e107-4708-9ed3-073af8fb8cb6" path="/var/lib/kubelet/pods/af39f530-e107-4708-9ed3-073af8fb8cb6/volumes"
Oct 30 03:27:19.730171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.730133    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"]
Oct 30 03:27:19.730719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:19.730681    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" containerID="cri-o://fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5" gracePeriod=30
Oct 30 03:27:21.438493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:21.438459    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:27:21.438947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:21.438520    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:27:25.019966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.019898    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:27:25.035533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.035497    4877 generic.go:356] "Generic (PLEG): container finished" podID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerID="fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5" exitCode=0
Oct 30 03:27:25.035701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.035560    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"
Oct 30 03:27:25.035701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.035582    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" event={"ID":"bfa8774c-c189-4cbf-81a1-b3466cc1250f","Type":"ContainerDied","Data":"fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5"}
Oct 30 03:27:25.035701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.035638    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk" event={"ID":"bfa8774c-c189-4cbf-81a1-b3466cc1250f","Type":"ContainerDied","Data":"da26fbc5e58cee6fc88b50538b47f97a06ed9154cd88b4e0d0433529e421a9f5"}
Oct 30 03:27:25.035701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.035662    4877 scope.go:122] "RemoveContainer" containerID="fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5"
Oct 30 03:27:25.045504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.045478    4877 scope.go:122] "RemoveContainer" containerID="b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"
Oct 30 03:27:25.054778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.054755    4877 scope.go:122] "RemoveContainer" containerID="fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5"
Oct 30 03:27:25.055056 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:27:25.055034    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5\": container with ID starting with fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5 not found: ID does not exist" containerID="fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5"
Oct 30 03:27:25.055145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.055064    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5"} err="failed to get container status \"fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5\": rpc error: code = NotFound desc = could not find container \"fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5\": container with ID starting with fb59189e023a52f3c7e322498a0278a12858a677286ea54f590f4989294333b5 not found: ID does not exist"
Oct 30 03:27:25.055145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.055084    4877 scope.go:122] "RemoveContainer" containerID="b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"
Oct 30 03:27:25.055320 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:27:25.055303    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3\": container with ID starting with b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3 not found: ID does not exist" containerID="b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"
Oct 30 03:27:25.055364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.055327    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3"} err="failed to get container status \"b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3\": rpc error: code = NotFound desc = could not find container \"b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3\": container with ID starting with b20b2a59c6a0b29163c680ea609d264059cb479a0650ab241bb940606f3c46a3 not found: ID does not exist"
Oct 30 03:27:25.075984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.075957    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076010    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076060    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076168    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076224    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076231    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:25.076358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076289    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076355    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076382    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076408    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:25.076535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076416    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data\") pod \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\" (UID: \"bfa8774c-c189-4cbf-81a1-b3466cc1250f\") "
Oct 30 03:27:25.076535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076453    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076697    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.076905    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.077124    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.077144    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.077158    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.077170    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.077217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.077182    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.078491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.078461    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:25.078980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.078954    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "kube-api-access-8fvlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:25.079066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.078977    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:25.079066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.079017    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy" pod "bfa8774c-c189-4cbf-81a1-b3466cc1250f" (UID: "bfa8774c-c189-4cbf-81a1-b3466cc1250f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:25.178603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.178571    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.178603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.178600    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.178603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.178610    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvlb\" (UniqueName: \"kubernetes.io/projected/bfa8774c-c189-4cbf-81a1-b3466cc1250f-kube-api-access-8fvlb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.178861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.178619    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa8774c-c189-4cbf-81a1-b3466cc1250f-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:25.386553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.386509    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"]
Oct 30 03:27:25.390460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.390419    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-91a20d-openshift-default-6c5dcdc445-mskkk"]
Oct 30 03:27:25.700254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:25.700170    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" path="/var/lib/kubelet/pods/bfa8774c-c189-4cbf-81a1-b3466cc1250f/volumes"
Oct 30 03:27:26.758462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:26.758416    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5_3ad05ccb-8d11-4718-826b-5d05bcabe7a1/payload-processing/0.log"
Oct 30 03:27:35.908813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.908776    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"]
Oct 30 03:27:35.909623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909598    4877 cpu_manager.go:414] "containerMap: removing container" podUID="af39f530-e107-4708-9ed3-073af8fb8cb6" containerName="payload-processing"
Oct 30 03:27:35.909623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909623    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af39f530-e107-4708-9ed3-073af8fb8cb6" containerName="payload-processing"
Oct 30 03:27:35.909805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909660    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy"
Oct 30 03:27:35.909805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909669    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy"
Oct 30 03:27:35.909805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909734    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4d8366e7-8e68-4622-912a-836f734e332f" containerName="payload-pre-processing"
Oct 30 03:27:35.909805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909742    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8366e7-8e68-4622-912a-836f734e332f" containerName="payload-pre-processing"
Oct 30 03:27:35.910009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909884    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy"
Oct 30 03:27:35.910009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909908    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8366e7-8e68-4622-912a-836f734e332f" containerName="payload-pre-processing"
Oct 30 03:27:35.910009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.909921    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="af39f530-e107-4708-9ed3-073af8fb8cb6" containerName="payload-processing"
Oct 30 03:27:35.915085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.915063    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:35.932816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.932787    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"]
Oct 30 03:27:35.938628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.938581    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:35.938911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.938886    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:27:35.963406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.963370    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:35.963626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.963500    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:35.963626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:35.963555    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9bx\" (UniqueName: \"kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.064338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.064299    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.064548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.064380    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9bx\" (UniqueName: \"kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.064548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.064533    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.065170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.065151    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.066897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.066871    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.083544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.083469    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9bx\" (UniqueName: \"kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx\") pod \"maas-api-revoke-sub-eb8314ed-nrprr\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.237106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.237071    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:36.545349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:36.545323    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"]
Oct 30 03:27:36.547455 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:27:36.547407    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cbaf32_7556_4afd_96e4_dfd19b95d866.slice/crio-be339b813e2b85277ee8b62b9fc16bfd5817329fb182de57f38d571dfc3b4a33 WatchSource:0}: Error finding container be339b813e2b85277ee8b62b9fc16bfd5817329fb182de57f38d571dfc3b4a33: Status 404 returned error can't find the container with id be339b813e2b85277ee8b62b9fc16bfd5817329fb182de57f38d571dfc3b4a33
Oct 30 03:27:37.163250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:37.163152    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr" event={"ID":"f3cbaf32-7556-4afd-96e4-dfd19b95d866","Type":"ContainerStarted","Data":"5c9996afb7434b58a6aff9cb4de3e4ac5a1e7880799a01cd14d77faede4bbc86"}
Oct 30 03:27:37.163250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:37.163223    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr" event={"ID":"f3cbaf32-7556-4afd-96e4-dfd19b95d866","Type":"ContainerStarted","Data":"be339b813e2b85277ee8b62b9fc16bfd5817329fb182de57f38d571dfc3b4a33"}
Oct 30 03:27:37.204510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:37.204413    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr" podStartSLOduration=2.204395528 podStartE2EDuration="2.204395528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:37.201803007 +0000 UTC m=+1485.630692429" watchObservedRunningTime="2026-10-30 03:27:37.204395528 +0000 UTC m=+1485.633284946"
Oct 30 03:27:38.173474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:38.173414    4877 generic.go:356] "Generic (PLEG): container finished" podID="f3cbaf32-7556-4afd-96e4-dfd19b95d866" containerID="5c9996afb7434b58a6aff9cb4de3e4ac5a1e7880799a01cd14d77faede4bbc86" exitCode=0
Oct 30 03:27:38.173896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:38.173522    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr" event={"ID":"f3cbaf32-7556-4afd-96e4-dfd19b95d866","Type":"ContainerDied","Data":"5c9996afb7434b58a6aff9cb4de3e4ac5a1e7880799a01cd14d77faede4bbc86"}
Oct 30 03:27:39.361234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.361199    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:39.434996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.434913    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca\") pod \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") "
Oct 30 03:27:39.435169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.435063    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca\") pod \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") "
Oct 30 03:27:39.435169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.435162    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx\" (UniqueName: \"kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx\") pod \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\" (UID: \"f3cbaf32-7556-4afd-96e4-dfd19b95d866\") "
Oct 30 03:27:39.435331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.435303    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca" pod "f3cbaf32-7556-4afd-96e4-dfd19b95d866" (UID: "f3cbaf32-7556-4afd-96e4-dfd19b95d866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:39.435775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.435752    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.437366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.437337    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca" pod "f3cbaf32-7556-4afd-96e4-dfd19b95d866" (UID: "f3cbaf32-7556-4afd-96e4-dfd19b95d866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:39.437795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.437770    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx" pod "f3cbaf32-7556-4afd-96e4-dfd19b95d866" (UID: "f3cbaf32-7556-4afd-96e4-dfd19b95d866"). InnerVolumeSpecName "kube-api-access-vk9bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:39.537041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.537008    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cbaf32-7556-4afd-96e4-dfd19b95d866-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.537041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.537036    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk9bx\" (UniqueName: \"kubernetes.io/projected/f3cbaf32-7556-4afd-96e4-dfd19b95d866-kube-api-access-vk9bx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:39.542809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.542585    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": read tcp 10.133.0.2:47114->10.133.0.152:15021: read: connection reset by peer" start-of-body=
Oct 30 03:27:39.542809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:39.542640    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": read tcp 10.133.0.2:47114->10.133.0.152:15021: read: connection reset by peer"
Oct 30 03:27:40.192808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.192773    4877 generic.go:356] "Generic (PLEG): container finished" podID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerID="a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72" exitCode=0
Oct 30 03:27:40.193012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.192850    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" event={"ID":"0d9cf411-4e26-4c4c-9716-c7df36175bc2","Type":"ContainerDied","Data":"a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"}
Oct 30 03:27:40.193448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.193399    4877 scope.go:122] "RemoveContainer" containerID="a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"
Oct 30 03:27:40.194861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.194543    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr" event={"ID":"f3cbaf32-7556-4afd-96e4-dfd19b95d866","Type":"ContainerDied","Data":"be339b813e2b85277ee8b62b9fc16bfd5817329fb182de57f38d571dfc3b4a33"}
Oct 30 03:27:40.194861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.194585    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be339b813e2b85277ee8b62b9fc16bfd5817329fb182de57f38d571dfc3b4a33"
Oct 30 03:27:40.194861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.194638    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"
Oct 30 03:27:40.195210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.195177    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:40.195316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.195266    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:40.195366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:40.195312    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:41.214976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:41.214941    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" event={"ID":"0d9cf411-4e26-4c4c-9716-c7df36175bc2","Type":"ContainerStarted","Data":"5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0"}
Oct 30 03:27:42.110532 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.110495    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:27:42.110813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.110783    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:27:42.110898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.110849    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 03:27:42.669018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.668982    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"]
Oct 30 03:27:42.669895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.669873    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f3cbaf32-7556-4afd-96e4-dfd19b95d866" containerName="revoke-keys"
Oct 30 03:27:42.670058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.670045    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cbaf32-7556-4afd-96e4-dfd19b95d866" containerName="revoke-keys"
Oct 30 03:27:42.670187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.670167    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy"
Oct 30 03:27:42.670187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.670184    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy"
Oct 30 03:27:42.670383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.670371    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cbaf32-7556-4afd-96e4-dfd19b95d866" containerName="revoke-keys"
Oct 30 03:27:42.670475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.670389    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa8774c-c189-4cbf-81a1-b3466cc1250f" containerName="istio-proxy"
Oct 30 03:27:42.674375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.674355    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.691866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.691834    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6c106c-openshift-default-dockercfg-k7lsk\""
Oct 30 03:27:42.712877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.712846    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"]
Oct 30 03:27:42.801478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.796257    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.801711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.801569    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.801711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.801657    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.801848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.801758    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.801959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.801941    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.802067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.802015    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.802067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.802045    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.802174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.802076    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.802174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.802107    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwfg\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903152    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903202    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903221    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwfg\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903293    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903319    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903359    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903409    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903503    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.903854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.903542    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.904606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.904578    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.904824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.904764    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.904824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.904652    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.905016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.904895    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.905495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.905461    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.906721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.906689    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.907115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.907095    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.922921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.922856    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:42.923018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:42.922911    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwfg\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg\") pod \"e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") " pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:43.020001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.019959    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:43.110679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.110639    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:27:43.110880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.110717    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 03:27:43.361601 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:27:43.361571    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e18eab_f08a_4813_8cef_c13b0c627150.slice/crio-64649560bfc2fcfae3901beff8b32c4f316bfa1be6c5da0c71acc227cc5809c1 WatchSource:0}: Error finding container 64649560bfc2fcfae3901beff8b32c4f316bfa1be6c5da0c71acc227cc5809c1: Status 404 returned error can't find the container with id 64649560bfc2fcfae3901beff8b32c4f316bfa1be6c5da0c71acc227cc5809c1
Oct 30 03:27:43.361748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.361724    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"]
Oct 30 03:27:43.364194 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.364161    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:43.364306 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.364262    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:43.364376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:43.364308    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:27:44.110859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:44.110822    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:27:44.111510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:44.111468    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 03:27:44.239134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:44.239094    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" event={"ID":"72e18eab-f08a-4813-8cef-c13b0c627150","Type":"ContainerStarted","Data":"0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a"}
Oct 30 03:27:44.239312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:44.239153    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" event={"ID":"72e18eab-f08a-4813-8cef-c13b0c627150","Type":"ContainerStarted","Data":"64649560bfc2fcfae3901beff8b32c4f316bfa1be6c5da0c71acc227cc5809c1"}
Oct 30 03:27:45.061033 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:45.055201    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:45.061033 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:45.059499    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:27:45.061033 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:45.059557    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:27:45.116672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:45.116199    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:27:45.116672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:45.116263    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 03:27:46.020906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:46.020858    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:27:46.021179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:46.021147    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:27:46.114485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:46.114448    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:27:46.114770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:46.114742    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 03:27:46.251728 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:27:46.251691    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:27:47.020486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:47.020448    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:27:47.020673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:47.020503    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:27:47.114025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:47.113994    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:27:47.114486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:47.114464    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:27:47.115175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:47.115155    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:27:47.199445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:47.198043    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podStartSLOduration=5.198019952 podStartE2EDuration="5.198019952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:44.299148302 +0000 UTC m=+1492.728037735" watchObservedRunningTime="2026-10-30 03:27:47.198019952 +0000 UTC m=+1495.626909371"
Oct 30 03:27:48.020833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:48.020793    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:27:48.021239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:48.020857    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:27:49.020688 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:49.020652    4877 patch_prober.go:29] interesting pod/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 03:27:49.020872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:49.020711    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 03:27:50.036343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:50.036307    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:50.036907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:50.036695    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:50.038201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:50.038169    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:27:51.034389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:51.034356    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"]
Oct 30 03:27:51.045658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:51.045628    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce3839f-qnsqn"]
Oct 30 03:27:51.701909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:51.701874    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc14c9e-6636-4e47-8b22-5fdc45d142d8" path="/var/lib/kubelet/pods/0dc14c9e-6636-4e47-8b22-5fdc45d142d8/volumes"
Oct 30 03:27:52.123070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:52.123029    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:27:52.155733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:52.155694    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:27:53.142118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.142079    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"]
Oct 30 03:27:53.142585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.142368    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy" containerID="cri-o://80d8da0e70675c273c7f07ef6afcfa830c80dceca4aae24c0799bb0aac14f78c" gracePeriod=30
Oct 30 03:27:53.345452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.344779    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"]
Oct 30 03:27:53.353586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.353558    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.361658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.361630    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"]
Oct 30 03:27:53.367830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.367803    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:53.367984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.367964    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:27:53.422884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.422794    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.423018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.422941    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.423130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.423110    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdz95\" (UniqueName: \"kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.524119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.524084    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.524333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.524157    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.524333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.524264    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdz95\" (UniqueName: \"kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.524983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.524957    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.526681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.526656    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.543577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.543548    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdz95\" (UniqueName: \"kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95\") pod \"maas-api-revoke-sub-b158d8b6-jfc72\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.673698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.673611    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:53.994450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:53.988742    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"]
Oct 30 03:27:54.395808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:54.395764    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72" event={"ID":"e615aebb-ce42-48a3-9885-6cd9c6ec6059","Type":"ContainerStarted","Data":"b047ca7e92f040439c9afb39aa28b1ff5f1ce08aeb12c06656a560c1b952e193"}
Oct 30 03:27:54.396231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:54.395839    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72" event={"ID":"e615aebb-ce42-48a3-9885-6cd9c6ec6059","Type":"ContainerStarted","Data":"42f42bea9b31cfb6e347ad15638f6cbdcec2b22dd9ff22795f7c64e6dbe1f14f"}
Oct 30 03:27:54.449485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:54.449411    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72" podStartSLOduration=1.44939698 podStartE2EDuration="1.44939698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:54.443026339 +0000 UTC m=+1502.871915757" watchObservedRunningTime="2026-10-30 03:27:54.44939698 +0000 UTC m=+1502.878286398"
Oct 30 03:27:54.469860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:54.469831    4877 scope.go:122] "RemoveContainer" containerID="b4fb38a8a1f7341d6f04fd6c852124bc5c66ea2a079f02cfc08f18d1193d4a6b"
Oct 30 03:27:55.405954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:55.405915    4877 generic.go:356] "Generic (PLEG): container finished" podID="e615aebb-ce42-48a3-9885-6cd9c6ec6059" containerID="b047ca7e92f040439c9afb39aa28b1ff5f1ce08aeb12c06656a560c1b952e193" exitCode=0
Oct 30 03:27:55.406379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:55.406009    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72" event={"ID":"e615aebb-ce42-48a3-9885-6cd9c6ec6059","Type":"ContainerDied","Data":"b047ca7e92f040439c9afb39aa28b1ff5f1ce08aeb12c06656a560c1b952e193"}
Oct 30 03:27:56.617736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.617710    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:56.687234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.687150    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca\") pod \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") "
Oct 30 03:27:56.687403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.687269    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca\") pod \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") "
Oct 30 03:27:56.687403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.687313    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95\" (UniqueName: \"kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95\") pod \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\" (UID: \"e615aebb-ce42-48a3-9885-6cd9c6ec6059\") "
Oct 30 03:27:56.687737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.687711    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca" pod "e615aebb-ce42-48a3-9885-6cd9c6ec6059" (UID: "e615aebb-ce42-48a3-9885-6cd9c6ec6059"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:56.689982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.689947    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca" pod "e615aebb-ce42-48a3-9885-6cd9c6ec6059" (UID: "e615aebb-ce42-48a3-9885-6cd9c6ec6059"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:27:56.690873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.690843    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95" pod "e615aebb-ce42-48a3-9885-6cd9c6ec6059" (UID: "e615aebb-ce42-48a3-9885-6cd9c6ec6059"). InnerVolumeSpecName "kube-api-access-gdz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:56.789829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.789785    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:56.789829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.789821    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e615aebb-ce42-48a3-9885-6cd9c6ec6059-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:56.789829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:56.789836    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdz95\" (UniqueName: \"kubernetes.io/projected/e615aebb-ce42-48a3-9885-6cd9c6ec6059-kube-api-access-gdz95\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:57.424264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:57.424227    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"
Oct 30 03:27:57.424264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:57.424241    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72" event={"ID":"e615aebb-ce42-48a3-9885-6cd9c6ec6059","Type":"ContainerDied","Data":"42f42bea9b31cfb6e347ad15638f6cbdcec2b22dd9ff22795f7c64e6dbe1f14f"}
Oct 30 03:27:57.424542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:57.424332    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f42bea9b31cfb6e347ad15638f6cbdcec2b22dd9ff22795f7c64e6dbe1f14f"
Oct 30 03:27:58.435274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.435244    4877 generic.go:356] "Generic (PLEG): container finished" podID="530e5791-dc0a-4d50-80c7-61241a65b260" containerID="80d8da0e70675c273c7f07ef6afcfa830c80dceca4aae24c0799bb0aac14f78c" exitCode=0
Oct 30 03:27:58.435667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.435293    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" event={"ID":"530e5791-dc0a-4d50-80c7-61241a65b260","Type":"ContainerDied","Data":"80d8da0e70675c273c7f07ef6afcfa830c80dceca4aae24c0799bb0aac14f78c"}
Oct 30 03:27:58.435667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.435335    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2" event={"ID":"530e5791-dc0a-4d50-80c7-61241a65b260","Type":"ContainerDied","Data":"9c6142ddc2589d2b6bdbd11bb5a12180a590ea70b9cf0140ecf8784becff408c"}
Oct 30 03:27:58.435667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.435353    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c6142ddc2589d2b6bdbd11bb5a12180a590ea70b9cf0140ecf8784becff408c"
Oct 30 03:27:58.435667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.435366    4877 scope.go:122] "RemoveContainer" containerID="25b2a0290070718a91fd09016dbbe9193786065b0dd82a5f796f992a577b04f3"
Oct 30 03:27:58.442830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.442810    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:27:58.474025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.473998    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"]
Oct 30 03:27:58.474592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474579    4877 cpu_manager.go:414] "containerMap: removing container" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy"
Oct 30 03:27:58.474645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474593    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy"
Oct 30 03:27:58.474645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474612    4877 cpu_manager.go:414] "containerMap: removing container" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy"
Oct 30 03:27:58.474645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474618    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy"
Oct 30 03:27:58.474645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474629    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e615aebb-ce42-48a3-9885-6cd9c6ec6059" containerName="revoke-keys"
Oct 30 03:27:58.474645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474634    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e615aebb-ce42-48a3-9885-6cd9c6ec6059" containerName="revoke-keys"
Oct 30 03:27:58.474801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474730    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy"
Oct 30 03:27:58.474801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474738    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" containerName="istio-proxy"
Oct 30 03:27:58.474801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.474745    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e615aebb-ce42-48a3-9885-6cd9c6ec6059" containerName="revoke-keys"
Oct 30 03:27:58.479211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.479186    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.501255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.501221    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"]
Oct 30 03:27:58.504509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.504382    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:27:58.504509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.504464    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:27:58.517503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.517471    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520122    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.520280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520218    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520284    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520340    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520371    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520411    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520474    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520529    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.520717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.520636    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket\") pod \"530e5791-dc0a-4d50-80c7-61241a65b260\" (UID: \"530e5791-dc0a-4d50-80c7-61241a65b260\") "
Oct 30 03:27:58.522061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.521171    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.522061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.521448    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.522061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.521620    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.524461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.523908    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:27:58.524461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.523931    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "kube-api-access-9b4q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:58.524461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.524253    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.524922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.524774    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:27:58.526586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.526556    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:27:58.527316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.527291    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo" pod "530e5791-dc0a-4d50-80c7-61241a65b260" (UID: "530e5791-dc0a-4d50-80c7-61241a65b260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:27:58.622650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.622614    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.622882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.622851    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.622952 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.622912    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhdd\" (UniqueName: \"kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.623021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623006    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/530e5791-dc0a-4d50-80c7-61241a65b260-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623026    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/530e5791-dc0a-4d50-80c7-61241a65b260-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623035    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623044    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623052    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623061    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b4q7\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-kube-api-access-9b4q7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623071    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/530e5791-dc0a-4d50-80c7-61241a65b260-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.623188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.623083    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/530e5791-dc0a-4d50-80c7-61241a65b260-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:27:58.724638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.724550    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.724830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.724724    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.724830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.724768    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhdd\" (UniqueName: \"kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.725467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.725414    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.727149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.727124    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.747352 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.747322    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhdd\" (UniqueName: \"kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd\") pod \"maas-api-revoke-sub-7393321c-qkdlr\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:58.798195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:58.798153    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:27:59.107677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.107643    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"]
Oct 30 03:27:59.108642 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:27:59.108604    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f588f24_b25b_47ca_ac28_f1e2231d7c10.slice/crio-8b560ebb35c3a10c2d59ab0dfb810c14793321ffbb680b946f42f94c0b63e236 WatchSource:0}: Error finding container 8b560ebb35c3a10c2d59ab0dfb810c14793321ffbb680b946f42f94c0b63e236: Status 404 returned error can't find the container with id 8b560ebb35c3a10c2d59ab0dfb810c14793321ffbb680b946f42f94c0b63e236
Oct 30 03:27:59.444055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.444013    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"
Oct 30 03:27:59.446025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.445989    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr" event={"ID":"6f588f24-b25b-47ca-ac28-f1e2231d7c10","Type":"ContainerStarted","Data":"3dc6040855cdf83df67e57c7e742d6f8de8761d401adf6dab36e0e37ce38d2d5"}
Oct 30 03:27:59.446182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.446048    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr" event={"ID":"6f588f24-b25b-47ca-ac28-f1e2231d7c10","Type":"ContainerStarted","Data":"8b560ebb35c3a10c2d59ab0dfb810c14793321ffbb680b946f42f94c0b63e236"}
Oct 30 03:27:59.533897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.533842    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr" podStartSLOduration=1.533828937 podStartE2EDuration="1.533828937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:59.532181615 +0000 UTC m=+1507.961071036" watchObservedRunningTime="2026-10-30 03:27:59.533828937 +0000 UTC m=+1507.962718394"
Oct 30 03:27:59.569166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.569123    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"]
Oct 30 03:27:59.572477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.572451    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7ce670-openshift-default-7948fc4f65-8glh2"]
Oct 30 03:27:59.702143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:27:59.702052    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530e5791-dc0a-4d50-80c7-61241a65b260" path="/var/lib/kubelet/pods/530e5791-dc0a-4d50-80c7-61241a65b260/volumes"
Oct 30 03:28:00.472917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:00.472855    4877 generic.go:356] "Generic (PLEG): container finished" podID="6f588f24-b25b-47ca-ac28-f1e2231d7c10" containerID="3dc6040855cdf83df67e57c7e742d6f8de8761d401adf6dab36e0e37ce38d2d5" exitCode=0
Oct 30 03:28:00.472917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:00.472915    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr" event={"ID":"6f588f24-b25b-47ca-ac28-f1e2231d7c10","Type":"ContainerDied","Data":"3dc6040855cdf83df67e57c7e742d6f8de8761d401adf6dab36e0e37ce38d2d5"}
Oct 30 03:28:00.714702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:00.714650    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-226.ec2.internal" podUID="2c7f2093544cf1327cc6ce2b1dd9d362" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:28:01.658606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.658575    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:28:01.693414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.693383    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca\") pod \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") "
Oct 30 03:28:01.693636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.693589    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca\") pod \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") "
Oct 30 03:28:01.693700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.693635    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd\" (UniqueName: \"kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd\") pod \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\" (UID: \"6f588f24-b25b-47ca-ac28-f1e2231d7c10\") "
Oct 30 03:28:01.693848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.693824    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca" pod "6f588f24-b25b-47ca-ac28-f1e2231d7c10" (UID: "6f588f24-b25b-47ca-ac28-f1e2231d7c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:01.694107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.694082    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.696296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.696243    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd" pod "6f588f24-b25b-47ca-ac28-f1e2231d7c10" (UID: "6f588f24-b25b-47ca-ac28-f1e2231d7c10"). InnerVolumeSpecName "kube-api-access-4hhdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:01.696296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.696282    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca" pod "6f588f24-b25b-47ca-ac28-f1e2231d7c10" (UID: "6f588f24-b25b-47ca-ac28-f1e2231d7c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:01.795520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.795483    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f588f24-b25b-47ca-ac28-f1e2231d7c10-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.795520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:01.795516    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhdd\" (UniqueName: \"kubernetes.io/projected/6f588f24-b25b-47ca-ac28-f1e2231d7c10-kube-api-access-4hhdd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:02.491114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:02.491081    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"
Oct 30 03:28:02.491292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:02.491108    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr" event={"ID":"6f588f24-b25b-47ca-ac28-f1e2231d7c10","Type":"ContainerDied","Data":"8b560ebb35c3a10c2d59ab0dfb810c14793321ffbb680b946f42f94c0b63e236"}
Oct 30 03:28:02.491292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:02.491174    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b560ebb35c3a10c2d59ab0dfb810c14793321ffbb680b946f42f94c0b63e236"
Oct 30 03:28:06.390473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.390403    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"]
Oct 30 03:28:06.391104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.391085    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6f588f24-b25b-47ca-ac28-f1e2231d7c10" containerName="revoke-keys"
Oct 30 03:28:06.391155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.391104    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f588f24-b25b-47ca-ac28-f1e2231d7c10" containerName="revoke-keys"
Oct 30 03:28:06.391237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.391227    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f588f24-b25b-47ca-ac28-f1e2231d7c10" containerName="revoke-keys"
Oct 30 03:28:06.394807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.394789    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.429476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.423229    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:06.429476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.423330    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:28:06.429476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.427529    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"]
Oct 30 03:28:06.455169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.455128    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.455439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.455402    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5z9s\" (UniqueName: \"kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.455520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.455466    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.556859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.556816    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z9s\" (UniqueName: \"kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.557086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.556893    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.557086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.556940    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.557724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.557695    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.559671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.559640    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.576476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.576412    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5z9s\" (UniqueName: \"kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s\") pod \"maas-api-revoke-sub-7bbea6be-dsvhh\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:06.708308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:06.708224    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:07.027725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:07.027690    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"]
Oct 30 03:28:07.027984 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:07.027957    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6898126_b2a3_49cb_b21d_03073fbc7ea8.slice/crio-8c987613e5eee8fcdbe3554b4cad7874d53f633de040745dcb0beaeca3a5e057 WatchSource:0}: Error finding container 8c987613e5eee8fcdbe3554b4cad7874d53f633de040745dcb0beaeca3a5e057: Status 404 returned error can't find the container with id 8c987613e5eee8fcdbe3554b4cad7874d53f633de040745dcb0beaeca3a5e057
Oct 30 03:28:07.530235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:07.530183    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh" event={"ID":"f6898126-b2a3-49cb-b21d-03073fbc7ea8","Type":"ContainerStarted","Data":"e4198c6083ef4095d254296445ef1beb83f51a75465676cfd8eea3655f339002"}
Oct 30 03:28:07.530708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:07.530258    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh" event={"ID":"f6898126-b2a3-49cb-b21d-03073fbc7ea8","Type":"ContainerStarted","Data":"8c987613e5eee8fcdbe3554b4cad7874d53f633de040745dcb0beaeca3a5e057"}
Oct 30 03:28:07.569775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:07.569660    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh" podStartSLOduration=1.56964245 podStartE2EDuration="1.56964245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:07.567097034 +0000 UTC m=+1515.995986452" watchObservedRunningTime="2026-10-30 03:28:07.56964245 +0000 UTC m=+1515.998531865"
Oct 30 03:28:08.538952 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:08.538917    4877 generic.go:356] "Generic (PLEG): container finished" podID="f6898126-b2a3-49cb-b21d-03073fbc7ea8" containerID="e4198c6083ef4095d254296445ef1beb83f51a75465676cfd8eea3655f339002" exitCode=0
Oct 30 03:28:08.539334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:08.539003    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh" event={"ID":"f6898126-b2a3-49cb-b21d-03073fbc7ea8","Type":"ContainerDied","Data":"e4198c6083ef4095d254296445ef1beb83f51a75465676cfd8eea3655f339002"}
Oct 30 03:28:09.701414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.701385    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:09.802229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.802145    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca\") pod \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") "
Oct 30 03:28:09.802383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.802274    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s\" (UniqueName: \"kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s\") pod \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") "
Oct 30 03:28:09.802383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.802338    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca\") pod \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\" (UID: \"f6898126-b2a3-49cb-b21d-03073fbc7ea8\") "
Oct 30 03:28:09.802615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.802589    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca" pod "f6898126-b2a3-49cb-b21d-03073fbc7ea8" (UID: "f6898126-b2a3-49cb-b21d-03073fbc7ea8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:09.803246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.803221    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:09.805004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.804970    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca" pod "f6898126-b2a3-49cb-b21d-03073fbc7ea8" (UID: "f6898126-b2a3-49cb-b21d-03073fbc7ea8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:09.805102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.805004    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s" pod "f6898126-b2a3-49cb-b21d-03073fbc7ea8" (UID: "f6898126-b2a3-49cb-b21d-03073fbc7ea8"). InnerVolumeSpecName "kube-api-access-d5z9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:09.905223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.905187    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5z9s\" (UniqueName: \"kubernetes.io/projected/f6898126-b2a3-49cb-b21d-03073fbc7ea8-kube-api-access-d5z9s\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:09.905223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:09.905224    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6898126-b2a3-49cb-b21d-03073fbc7ea8-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:10.554359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:10.554330    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"
Oct 30 03:28:10.554562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:10.554372    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh" event={"ID":"f6898126-b2a3-49cb-b21d-03073fbc7ea8","Type":"ContainerDied","Data":"8c987613e5eee8fcdbe3554b4cad7874d53f633de040745dcb0beaeca3a5e057"}
Oct 30 03:28:10.554562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:10.554434    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c987613e5eee8fcdbe3554b4cad7874d53f633de040745dcb0beaeca3a5e057"
Oct 30 03:28:11.375451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.374605    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"]
Oct 30 03:28:11.375925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.375608    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f6898126-b2a3-49cb-b21d-03073fbc7ea8" containerName="revoke-keys"
Oct 30 03:28:11.375925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.375629    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6898126-b2a3-49cb-b21d-03073fbc7ea8" containerName="revoke-keys"
Oct 30 03:28:11.375925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.375785    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6898126-b2a3-49cb-b21d-03073fbc7ea8" containerName="revoke-keys"
Oct 30 03:28:11.389999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.389969    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.401548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.400862    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"]
Oct 30 03:28:11.415074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.415046    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:11.415445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.415400    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:28:11.429611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.429575    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.429772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.429705    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n52h\" (UniqueName: \"kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.429772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.429739    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.531178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.531142    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n52h\" (UniqueName: \"kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.531178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.531183    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.531374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.531326    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.531909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.531882    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.533470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.533448    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.550631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.550606    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n52h\" (UniqueName: \"kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h\") pod \"maas-api-revoke-sub-99c29f60-xlfmb\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:11.716218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:11.716122    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:12.036749 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:12.036712    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76a156c_822f_4067_9b42_1aa4643e1923.slice/crio-e1a2a4569fed0baa2c763332abcd29b6dd8a5e68916729fc8490fd4902e9b378 WatchSource:0}: Error finding container e1a2a4569fed0baa2c763332abcd29b6dd8a5e68916729fc8490fd4902e9b378: Status 404 returned error can't find the container with id e1a2a4569fed0baa2c763332abcd29b6dd8a5e68916729fc8490fd4902e9b378
Oct 30 03:28:12.037069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.037046    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"]
Oct 30 03:28:12.571012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.570975    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb" event={"ID":"c76a156c-822f-4067-9b42-1aa4643e1923","Type":"ContainerStarted","Data":"40f85f4f694054cbe4078719f358616bab57b2f44121e6c31d880c4256fd23c5"}
Oct 30 03:28:12.571490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.571053    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb" event={"ID":"c76a156c-822f-4067-9b42-1aa4643e1923","Type":"ContainerStarted","Data":"e1a2a4569fed0baa2c763332abcd29b6dd8a5e68916729fc8490fd4902e9b378"}
Oct 30 03:28:12.612851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.612796    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb" podStartSLOduration=1.612783082 podStartE2EDuration="1.612783082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:12.609178563 +0000 UTC m=+1521.038067985" watchObservedRunningTime="2026-10-30 03:28:12.612783082 +0000 UTC m=+1521.041672499"
Oct 30 03:28:12.803742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.803701    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"]
Oct 30 03:28:12.808345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.808292    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.827782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.827750    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a9c9b6-openshift-default-dockercfg-9jv62\""
Oct 30 03:28:12.830164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.830124    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"]
Oct 30 03:28:12.846894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.846867    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmx4\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.846916    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.846945    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.847040    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.847078    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.847110    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.847131    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.847162    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.847225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.847192    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948581    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmx4\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948643    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948671    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948719    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948772    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948822    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948855    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.948964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948913    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.949122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.948963    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.949122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.949106    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.949236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.949144    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.949666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.949636    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.949797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.949673    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.949797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.949705    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.951439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.951405    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.951682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.951662    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.967210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.967182    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:12.967517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:12.967499    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmx4\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:13.124480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.124378    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:13.464741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.463607    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"]
Oct 30 03:28:13.468966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.468931    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:13.469135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.469000    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:13.469135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.469030    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:13.585262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.585216    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" event={"ID":"9a26cda8-0568-4e08-af78-c239f8139068","Type":"ContainerStarted","Data":"c0bcd926818638e28f0ec60e586dd3b8645ad4b944b0ad4fdb44e759f6fb569c"}
Oct 30 03:28:13.585807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.585288    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" event={"ID":"9a26cda8-0568-4e08-af78-c239f8139068","Type":"ContainerStarted","Data":"7c76c4bf1e3c62a197eeb13d8a7aafd1a9494c7f9b7e3ce8bc9d9b7d088885f4"}
Oct 30 03:28:13.590350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.590323    4877 generic.go:356] "Generic (PLEG): container finished" podID="c76a156c-822f-4067-9b42-1aa4643e1923" containerID="40f85f4f694054cbe4078719f358616bab57b2f44121e6c31d880c4256fd23c5" exitCode=0
Oct 30 03:28:13.590753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.590647    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb" event={"ID":"c76a156c-822f-4067-9b42-1aa4643e1923","Type":"ContainerDied","Data":"40f85f4f694054cbe4078719f358616bab57b2f44121e6c31d880c4256fd23c5"}
Oct 30 03:28:13.681522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:13.681458    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podStartSLOduration=1.681417412 podStartE2EDuration="1.681417412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:13.637739855 +0000 UTC m=+1522.066629274" watchObservedRunningTime="2026-10-30 03:28:13.681417412 +0000 UTC m=+1522.110306832"
Oct 30 03:28:14.125447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:14.125337    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:14.135883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:14.135851    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:14.137419 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:14.137381    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:14.908117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:14.907716    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:15.027015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.026973    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca\") pod \"c76a156c-822f-4067-9b42-1aa4643e1923\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") "
Oct 30 03:28:15.027214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.027200    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca\") pod \"c76a156c-822f-4067-9b42-1aa4643e1923\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") "
Oct 30 03:28:15.030550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.029221    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h\" (UniqueName: \"kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h\") pod \"c76a156c-822f-4067-9b42-1aa4643e1923\" (UID: \"c76a156c-822f-4067-9b42-1aa4643e1923\") "
Oct 30 03:28:15.031332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.031280    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca" pod "c76a156c-822f-4067-9b42-1aa4643e1923" (UID: "c76a156c-822f-4067-9b42-1aa4643e1923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:15.032218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.031849    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca" pod "c76a156c-822f-4067-9b42-1aa4643e1923" (UID: "c76a156c-822f-4067-9b42-1aa4643e1923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:15.032218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.032161    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h" pod "c76a156c-822f-4067-9b42-1aa4643e1923" (UID: "c76a156c-822f-4067-9b42-1aa4643e1923"). InnerVolumeSpecName "kube-api-access-7n52h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:15.127060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.127021    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:15.127259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.127105    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:15.131143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.131070    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:15.131143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.131102    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c76a156c-822f-4067-9b42-1aa4643e1923-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:15.131143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.131119    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n52h\" (UniqueName: \"kubernetes.io/projected/c76a156c-822f-4067-9b42-1aa4643e1923-kube-api-access-7n52h\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:15.648622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.648576    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb" event={"ID":"c76a156c-822f-4067-9b42-1aa4643e1923","Type":"ContainerDied","Data":"e1a2a4569fed0baa2c763332abcd29b6dd8a5e68916729fc8490fd4902e9b378"}
Oct 30 03:28:15.648934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.648913    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a2a4569fed0baa2c763332abcd29b6dd8a5e68916729fc8490fd4902e9b378"
Oct 30 03:28:15.649091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:15.648609    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"
Oct 30 03:28:16.124769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:16.124737    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:16.125192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:16.124790    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:16.234963 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:16.234922    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:28:17.125526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:17.125484    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:17.126036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:17.125554    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:18.125766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:18.125723    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:18.126255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:18.125796    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:19.125989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.125952    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:19.127000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.126586    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:19.420009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.419886    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"]
Oct 30 03:28:19.421247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.421213    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c76a156c-822f-4067-9b42-1aa4643e1923" containerName="revoke-keys"
Oct 30 03:28:19.421413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.421398    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76a156c-822f-4067-9b42-1aa4643e1923" containerName="revoke-keys"
Oct 30 03:28:19.421752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.421734    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76a156c-822f-4067-9b42-1aa4643e1923" containerName="revoke-keys"
Oct 30 03:28:19.430689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.430668    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.456103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.456070    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"]
Oct 30 03:28:19.461678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.461632    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:19.461800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.461701    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:28:19.609873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.609832    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.610045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.609985    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9m2\" (UniqueName: \"kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.610121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.610099    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.710903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.710814    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.711079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.710904    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9m2\" (UniqueName: \"kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.711079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.710954    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.711833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.711806    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.714057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.714020    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.730874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.730846    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9m2\" (UniqueName: \"kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2\") pod \"maas-api-revoke-sub-ee60a867-zr8px\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:19.765804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:19.765777    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:20.089821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.089791    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"]
Oct 30 03:28:20.091237 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:20.091196    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0f9141_2238_4a84_bf8b_45c1d15f2ad8.slice/crio-c689b1d61e2a5317d3928a693e62296df5da7828f225864d465bcb5e9bfccbdd WatchSource:0}: Error finding container c689b1d61e2a5317d3928a693e62296df5da7828f225864d465bcb5e9bfccbdd: Status 404 returned error can't find the container with id c689b1d61e2a5317d3928a693e62296df5da7828f225864d465bcb5e9bfccbdd
Oct 30 03:28:20.125231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.125205    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 03:28:20.125359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.125258    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 03:28:20.187729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.187697    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"]
Oct 30 03:28:20.188153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.187923    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy" containerID="cri-o://2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b" gracePeriod=30
Oct 30 03:28:20.202903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.202863    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"]
Oct 30 03:28:20.203356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.203319    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy" containerID="cri-o://42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2" gracePeriod=30
Oct 30 03:28:20.209026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.208994    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"]
Oct 30 03:28:20.209344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.209274    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy" containerID="cri-o://b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234" gracePeriod=30
Oct 30 03:28:20.697381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.697334    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px" event={"ID":"db0f9141-2238-4a84-bf8b-45c1d15f2ad8","Type":"ContainerStarted","Data":"f3bcadf41176523ef4113ba0d1b803cdeab216d510bd242e053f89231a33caa2"}
Oct 30 03:28:20.697626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.697399    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px" event={"ID":"db0f9141-2238-4a84-bf8b-45c1d15f2ad8","Type":"ContainerStarted","Data":"c689b1d61e2a5317d3928a693e62296df5da7828f225864d465bcb5e9bfccbdd"}
Oct 30 03:28:20.738110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:20.738004    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px" podStartSLOduration=1.737991236 podStartE2EDuration="1.737991236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:20.735013154 +0000 UTC m=+1529.163902572" watchObservedRunningTime="2026-10-30 03:28:20.737991236 +0000 UTC m=+1529.166880654"
Oct 30 03:28:21.710097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:21.710053    4877 generic.go:356] "Generic (PLEG): container finished" podID="db0f9141-2238-4a84-bf8b-45c1d15f2ad8" containerID="f3bcadf41176523ef4113ba0d1b803cdeab216d510bd242e053f89231a33caa2" exitCode=0
Oct 30 03:28:21.710551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:21.710112    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px" event={"ID":"db0f9141-2238-4a84-bf8b-45c1d15f2ad8","Type":"ContainerDied","Data":"f3bcadf41176523ef4113ba0d1b803cdeab216d510bd242e053f89231a33caa2"}
Oct 30 03:28:21.839654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:21.839617    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:21.840210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:21.840188    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:21.841168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:21.841143    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:28:22.905077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.905051    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:22.953911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.953879    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca\") pod \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") "
Oct 30 03:28:22.954103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.953967    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca\") pod \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") "
Oct 30 03:28:22.954167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.954140    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2\" (UniqueName: \"kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2\") pod \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\" (UID: \"db0f9141-2238-4a84-bf8b-45c1d15f2ad8\") "
Oct 30 03:28:22.954258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.954237    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca" pod "db0f9141-2238-4a84-bf8b-45c1d15f2ad8" (UID: "db0f9141-2238-4a84-bf8b-45c1d15f2ad8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:22.954731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.954695    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:22.957221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.957193    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca" pod "db0f9141-2238-4a84-bf8b-45c1d15f2ad8" (UID: "db0f9141-2238-4a84-bf8b-45c1d15f2ad8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:22.957321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:22.957256    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2" pod "db0f9141-2238-4a84-bf8b-45c1d15f2ad8" (UID: "db0f9141-2238-4a84-bf8b-45c1d15f2ad8"). InnerVolumeSpecName "kube-api-access-sd9m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:23.056365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.056329    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:23.056365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.056361    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9m2\" (UniqueName: \"kubernetes.io/projected/db0f9141-2238-4a84-bf8b-45c1d15f2ad8-kube-api-access-sd9m2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:23.726501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.726465    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"
Oct 30 03:28:23.726501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.726486    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px" event={"ID":"db0f9141-2238-4a84-bf8b-45c1d15f2ad8","Type":"ContainerDied","Data":"c689b1d61e2a5317d3928a693e62296df5da7828f225864d465bcb5e9bfccbdd"}
Oct 30 03:28:23.726775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.726542    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c689b1d61e2a5317d3928a693e62296df5da7828f225864d465bcb5e9bfccbdd"
Oct 30 03:28:23.772448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.772384    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"]
Oct 30 03:28:23.772680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:23.772646    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy" containerID="cri-o://9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1" gracePeriod=30
Oct 30 03:28:25.522912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.521578    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:28:25.558435 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.558405    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:28:25.572305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.572281    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:28:25.597391 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597352    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.597599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597468    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.597599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597513    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.597599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597558    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.597811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597597    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.597811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597647    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.597926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597842    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.597987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597948    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.598044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.597987    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:25.598044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598035    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.598156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598068    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.598156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598114    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598156    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.598266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598194    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.598375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598290    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.598375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598337    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.598375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598349    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.598375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598370    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598416    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598466    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.598626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598501    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.598626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598550    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598591    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.598898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598645    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598687    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.598898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598781    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598833    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.598898 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598872    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.599163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598910    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token\") pod \"79415bef-b337-4492-9ab8-3c3081ce7574\" (UID: \"79415bef-b337-4492-9ab8-3c3081ce7574\") "
Oct 30 03:28:25.599163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598949    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket\") pod \"79fea061-7b0d-4148-84cf-e82942c93082\" (UID: \"79fea061-7b0d-4148-84cf-e82942c93082\") "
Oct 30 03:28:25.599163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.598998    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy\") pod \"ebbe6556-4143-44df-b4a0-232d5971626f\" (UID: \"ebbe6556-4143-44df-b4a0-232d5971626f\") "
Oct 30 03:28:25.599323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.599173    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.599793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.599685    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.599793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.599710    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.599793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.599726    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.599793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.599740    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.600109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.600036    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.600475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.600253    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.600610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.600583    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:25.600686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.600640    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.601103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.600844    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.601103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.600937    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.601433 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.601304    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:25.601575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.601552    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.601646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.601596    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:25.601907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.601884    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.603152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.603126    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:25.604389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.604364    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:25.606605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.606577    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.608867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.608809    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:25.608867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.608838    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "kube-api-access-m7c68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:25.608867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.608842    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.608867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.608844    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "kube-api-access-mg6cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:25.609308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.609215    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:25.609308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.609249    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy" pod "79fea061-7b0d-4148-84cf-e82942c93082" (UID: "79fea061-7b0d-4148-84cf-e82942c93082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.609308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.609256    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:25.609308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.609277    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token" pod "ebbe6556-4143-44df-b4a0-232d5971626f" (UID: "ebbe6556-4143-44df-b4a0-232d5971626f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:25.611383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.611359    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd" pod "79415bef-b337-4492-9ab8-3c3081ce7574" (UID: "79415bef-b337-4492-9ab8-3c3081ce7574"). InnerVolumeSpecName "kube-api-access-xj4hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:25.700867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.700830    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.700867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.700867    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79415bef-b337-4492-9ab8-3c3081ce7574-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.700881    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.700980    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.700994    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701010    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701023    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbe6556-4143-44df-b4a0-232d5971626f-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701036    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701053    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbe6556-4143-44df-b4a0-232d5971626f-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701069    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fea061-7b0d-4148-84cf-e82942c93082-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701083    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6cx\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-kube-api-access-mg6cx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701096    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701111    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbe6556-4143-44df-b4a0-232d5971626f-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701124    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c68\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-kube-api-access-m7c68\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701137    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701151    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbe6556-4143-44df-b4a0-232d5971626f-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701164    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79415bef-b337-4492-9ab8-3c3081ce7574-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701178    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701194    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fea061-7b0d-4148-84cf-e82942c93082-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701206    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj4hd\" (UniqueName: \"kubernetes.io/projected/79415bef-b337-4492-9ab8-3c3081ce7574-kube-api-access-xj4hd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701220    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fea061-7b0d-4148-84cf-e82942c93082-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701237    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79415bef-b337-4492-9ab8-3c3081ce7574-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.701456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.701248    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fea061-7b0d-4148-84cf-e82942c93082-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:25.742256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.742167    4877 generic.go:356] "Generic (PLEG): container finished" podID="79415bef-b337-4492-9ab8-3c3081ce7574" containerID="2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b" exitCode=0
Oct 30 03:28:25.742256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.742217    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"
Oct 30 03:28:25.742489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.742249    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" event={"ID":"79415bef-b337-4492-9ab8-3c3081ce7574","Type":"ContainerDied","Data":"2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b"}
Oct 30 03:28:25.742489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.742308    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44" event={"ID":"79415bef-b337-4492-9ab8-3c3081ce7574","Type":"ContainerDied","Data":"1e0e262c3e15f6b97580b91a03d1124550684d4c2910bfdd54e67d6acac9e63b"}
Oct 30 03:28:25.742489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.742330    4877 scope.go:122] "RemoveContainer" containerID="2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b"
Oct 30 03:28:25.743920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.743902    4877 generic.go:356] "Generic (PLEG): container finished" podID="ebbe6556-4143-44df-b4a0-232d5971626f" containerID="42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2" exitCode=0
Oct 30 03:28:25.744023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.743961    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"
Oct 30 03:28:25.744023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.743998    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" event={"ID":"ebbe6556-4143-44df-b4a0-232d5971626f","Type":"ContainerDied","Data":"42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2"}
Oct 30 03:28:25.744139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.744038    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t" event={"ID":"ebbe6556-4143-44df-b4a0-232d5971626f","Type":"ContainerDied","Data":"ed200e1485f0169aff5c2f0b8938f1d4c39b249ece47a487ed4ff1bfe820ca83"}
Oct 30 03:28:25.746254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.746231    4877 generic.go:356] "Generic (PLEG): container finished" podID="79fea061-7b0d-4148-84cf-e82942c93082" containerID="b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234" exitCode=0
Oct 30 03:28:25.746584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.746282    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"
Oct 30 03:28:25.746584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.746323    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" event={"ID":"79fea061-7b0d-4148-84cf-e82942c93082","Type":"ContainerDied","Data":"b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234"}
Oct 30 03:28:25.746584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.746384    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj" event={"ID":"79fea061-7b0d-4148-84cf-e82942c93082","Type":"ContainerDied","Data":"c0e0ed253213a99b14ca138c011c8d40e18b5dd1cbb3151ba64cafbf7f744f78"}
Oct 30 03:28:25.754690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.754673    4877 scope.go:122] "RemoveContainer" containerID="2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b"
Oct 30 03:28:25.754969 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:25.754949    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b\": container with ID starting with 2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b not found: ID does not exist" containerID="2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b"
Oct 30 03:28:25.755030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.754976    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b"} err="failed to get container status \"2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b\": rpc error: code = NotFound desc = could not find container \"2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b\": container with ID starting with 2dc210e63a8032195c5a602cd4306b4ce1872ecc7d20f02255354ac3320bb44b not found: ID does not exist"
Oct 30 03:28:25.755030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.754992    4877 scope.go:122] "RemoveContainer" containerID="42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2"
Oct 30 03:28:25.766375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.766310    4877 scope.go:122] "RemoveContainer" containerID="42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2"
Oct 30 03:28:25.767538 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:25.767512    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2\": container with ID starting with 42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2 not found: ID does not exist" containerID="42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2"
Oct 30 03:28:25.767711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.767683    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2"} err="failed to get container status \"42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2\": rpc error: code = NotFound desc = could not find container \"42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2\": container with ID starting with 42abd6bbeff0a1c6d0adee23f0b3e7c05d54e3dbf44f3e4396d5120feb145db2 not found: ID does not exist"
Oct 30 03:28:25.767847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.767811    4877 scope.go:122] "RemoveContainer" containerID="b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234"
Oct 30 03:28:25.778320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.778298    4877 scope.go:122] "RemoveContainer" containerID="b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234"
Oct 30 03:28:25.778638 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:25.778616    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234\": container with ID starting with b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234 not found: ID does not exist" containerID="b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234"
Oct 30 03:28:25.778721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.778645    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234"} err="failed to get container status \"b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234\": rpc error: code = NotFound desc = could not find container \"b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234\": container with ID starting with b854c1df42ea8dbba18fd5d29ba6bf6ddcc0a53a6c4cc6cfca9e9f9a8a64b234 not found: ID does not exist"
Oct 30 03:28:25.797764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.797734    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"]
Oct 30 03:28:25.804281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.804245    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r9z2t"]
Oct 30 03:28:25.839139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.839101    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"]
Oct 30 03:28:25.844989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.844954    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6vq44"]
Oct 30 03:28:25.876746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.876713    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"]
Oct 30 03:28:25.884980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:25.884947    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hcwvj"]
Oct 30 03:28:27.700358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:27.700327    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" path="/var/lib/kubelet/pods/79415bef-b337-4492-9ab8-3c3081ce7574/volumes"
Oct 30 03:28:27.700876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:27.700864    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79fea061-7b0d-4148-84cf-e82942c93082" path="/var/lib/kubelet/pods/79fea061-7b0d-4148-84cf-e82942c93082/volumes"
Oct 30 03:28:27.701334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:27.701317    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" path="/var/lib/kubelet/pods/ebbe6556-4143-44df-b4a0-232d5971626f/volumes"
Oct 30 03:28:29.062486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.062413    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:28:29.145091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145056    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145132    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145160    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145180    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145224    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145262    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145330    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145364    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145479    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo\") pod \"8a95bd50-498b-4499-bd1e-e0b22b72de18\" (UID: \"8a95bd50-498b-4499-bd1e-e0b22b72de18\") "
Oct 30 03:28:29.145600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145514    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:29.145801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145710    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:29.145801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145730    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:29.146193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.145965    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:29.146295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.146044    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.146295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.146237    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a95bd50-498b-4499-bd1e-e0b22b72de18-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.146295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.146256    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.146295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.146158    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:29.148025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.147995    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:29.148123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.148046    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:29.148442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.148135    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:29.149148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.149123    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd" pod "8a95bd50-498b-4499-bd1e-e0b22b72de18" (UID: "8a95bd50-498b-4499-bd1e-e0b22b72de18"). InnerVolumeSpecName "kube-api-access-sv4fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:29.247214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.247174    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.247214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.247205    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.247453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.247227    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.247453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.247236    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.247453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.247244    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a95bd50-498b-4499-bd1e-e0b22b72de18-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.247453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.247254    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4fd\" (UniqueName: \"kubernetes.io/projected/8a95bd50-498b-4499-bd1e-e0b22b72de18-kube-api-access-sv4fd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:29.779476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.779412    4877 generic.go:356] "Generic (PLEG): container finished" podID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerID="9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1" exitCode=0
Oct 30 03:28:29.779684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.779495    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"
Oct 30 03:28:29.779684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.779500    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" event={"ID":"8a95bd50-498b-4499-bd1e-e0b22b72de18","Type":"ContainerDied","Data":"9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1"}
Oct 30 03:28:29.779684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.779574    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m" event={"ID":"8a95bd50-498b-4499-bd1e-e0b22b72de18","Type":"ContainerDied","Data":"c1637cfea135887f411fa364ee5acb3b91c7484ff77165447b936c50134a20df"}
Oct 30 03:28:29.779684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.779606    4877 scope.go:122] "RemoveContainer" containerID="9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1"
Oct 30 03:28:29.790394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.790372    4877 scope.go:122] "RemoveContainer" containerID="bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"
Oct 30 03:28:29.806166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.806141    4877 scope.go:122] "RemoveContainer" containerID="9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1"
Oct 30 03:28:29.806682 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:29.806658    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1\": container with ID starting with 9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1 not found: ID does not exist" containerID="9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1"
Oct 30 03:28:29.806811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.806692    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1"} err="failed to get container status \"9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1\": rpc error: code = NotFound desc = could not find container \"9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1\": container with ID starting with 9119c5dcefc5108283c2bbbce2c9409d458e949e45108e6f7c330b012c5019a1 not found: ID does not exist"
Oct 30 03:28:29.806811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.806712    4877 scope.go:122] "RemoveContainer" containerID="bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"
Oct 30 03:28:29.807021 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:29.806994    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73\": container with ID starting with bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73 not found: ID does not exist" containerID="bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"
Oct 30 03:28:29.807081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.807026    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73"} err="failed to get container status \"bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73\": rpc error: code = NotFound desc = could not find container \"bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73\": container with ID starting with bca79230efedc5ca63c77a937913217a8d813289e3d5b5fd0d68bac468b57b73 not found: ID does not exist"
Oct 30 03:28:29.838991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.838965    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"]
Oct 30 03:28:29.839559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839535    4877 cpu_manager.go:414] "containerMap: removing container" podUID="db0f9141-2238-4a84-bf8b-45c1d15f2ad8" containerName="revoke-keys"
Oct 30 03:28:29.839559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839552    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0f9141-2238-4a84-bf8b-45c1d15f2ad8" containerName="revoke-keys"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839571    4877 cpu_manager.go:414] "containerMap: removing container" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839578    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839591    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839598    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839611    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839618    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839641    4877 cpu_manager.go:414] "containerMap: removing container" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839649    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839660    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy"
Oct 30 03:28:29.839746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839668    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy"
Oct 30 03:28:29.840305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839904    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0f9141-2238-4a84-bf8b-45c1d15f2ad8" containerName="revoke-keys"
Oct 30 03:28:29.840305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839919    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="79fea061-7b0d-4148-84cf-e82942c93082" containerName="istio-proxy"
Oct 30 03:28:29.840305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839929    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy"
Oct 30 03:28:29.840305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839943    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" containerName="istio-proxy"
Oct 30 03:28:29.840305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839956    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="79415bef-b337-4492-9ab8-3c3081ce7574" containerName="istio-proxy"
Oct 30 03:28:29.840305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.839964    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbe6556-4143-44df-b4a0-232d5971626f" containerName="istio-proxy"
Oct 30 03:28:29.844652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.844633    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"]
Oct 30 03:28:29.844768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.844744    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:29.851333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.851309    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad5839-openshift-default-ff4b8c565-z726m"]
Oct 30 03:28:29.859459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.859434    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:29.863696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.863646    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:28:29.863696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.863678    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"]
Oct 30 03:28:29.953961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.953925    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2j2k\" (UniqueName: \"kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:29.954146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.953997    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:29.954146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:29.954057    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.055716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.055681    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.055890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.055730    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.056071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.056042    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2j2k\" (UniqueName: \"kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.056527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.056500    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.058297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.058277    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.074097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.074074    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2j2k\" (UniqueName: \"kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k\") pod \"maas-api-revoke-sub-aad9f014-g47xh\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.160140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.160099    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:30.470962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.470935    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"]
Oct 30 03:28:30.471479 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:30.471455    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e55ebd_8a02_4595_bcc1_d3625030f099.slice/crio-d3c8ad500e6b3c547c58b524f20592b4ab625c9fb15d5a328129d7e4e30f5dc8 WatchSource:0}: Error finding container d3c8ad500e6b3c547c58b524f20592b4ab625c9fb15d5a328129d7e4e30f5dc8: Status 404 returned error can't find the container with id d3c8ad500e6b3c547c58b524f20592b4ab625c9fb15d5a328129d7e4e30f5dc8
Oct 30 03:28:30.793954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.793889    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh" event={"ID":"a9e55ebd-8a02-4595-bcc1-d3625030f099","Type":"ContainerStarted","Data":"d3c8ad500e6b3c547c58b524f20592b4ab625c9fb15d5a328129d7e4e30f5dc8"}
Oct 30 03:28:30.936577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.936536    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"]
Oct 30 03:28:30.936788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.936751    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" podUID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" containerName="payload-processing" containerID="cri-o://7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1" gracePeriod=30
Oct 30 03:28:30.954729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.954699    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"]
Oct 30 03:28:30.954958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.954917    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" podUID="ada11452-b43e-4475-ab9e-321aa368f965" containerName="payload-pre-processing" containerID="cri-o://21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29" gracePeriod=30
Oct 30 03:28:30.960100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.960067    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" podUID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.142:9004: connect: connection refused"
Oct 30 03:28:30.962812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:30.962784    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" podUID="ada11452-b43e-4475-ab9e-321aa368f965" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.141:9004: connect: connection refused"
Oct 30 03:28:31.327088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.327001    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:28:31.328850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.328823    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:28:31.375959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.375927    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume\") pod \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") "
Oct 30 03:28:31.376129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.375993    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt\" (UniqueName: \"kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt\") pod \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\" (UID: \"3ad05ccb-8d11-4718-826b-5d05bcabe7a1\") "
Oct 30 03:28:31.376415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.376384    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume" pod "3ad05ccb-8d11-4718-826b-5d05bcabe7a1" (UID: "3ad05ccb-8d11-4718-826b-5d05bcabe7a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:31.376561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.376541    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:31.378670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.378630    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt" pod "3ad05ccb-8d11-4718-826b-5d05bcabe7a1" (UID: "3ad05ccb-8d11-4718-826b-5d05bcabe7a1"). InnerVolumeSpecName "kube-api-access-5zblt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:31.477762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.477726    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume\") pod \"ada11452-b43e-4475-ab9e-321aa368f965\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") "
Oct 30 03:28:31.477964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.477792    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74\") pod \"ada11452-b43e-4475-ab9e-321aa368f965\" (UID: \"ada11452-b43e-4475-ab9e-321aa368f965\") "
Oct 30 03:28:31.478201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.478171    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume" pod "ada11452-b43e-4475-ab9e-321aa368f965" (UID: "ada11452-b43e-4475-ab9e-321aa368f965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:31.478408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.478386    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zblt\" (UniqueName: \"kubernetes.io/projected/3ad05ccb-8d11-4718-826b-5d05bcabe7a1-kube-api-access-5zblt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:31.478502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.478413    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada11452-b43e-4475-ab9e-321aa368f965-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:31.480095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.480068    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74" pod "ada11452-b43e-4475-ab9e-321aa368f965" (UID: "ada11452-b43e-4475-ab9e-321aa368f965"). InnerVolumeSpecName "kube-api-access-9zx74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:31.580028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.579942    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zx74\" (UniqueName: \"kubernetes.io/projected/ada11452-b43e-4475-ab9e-321aa368f965-kube-api-access-9zx74\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:31.706479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.706410    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a95bd50-498b-4499-bd1e-e0b22b72de18" path="/var/lib/kubelet/pods/8a95bd50-498b-4499-bd1e-e0b22b72de18/volumes"
Oct 30 03:28:31.805418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.805369    4877 generic.go:356] "Generic (PLEG): container finished" podID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" containerID="7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1" exitCode=0
Oct 30 03:28:31.805418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.805415    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" event={"ID":"3ad05ccb-8d11-4718-826b-5d05bcabe7a1","Type":"ContainerDied","Data":"7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1"}
Oct 30 03:28:31.805710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.805500    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5" event={"ID":"3ad05ccb-8d11-4718-826b-5d05bcabe7a1","Type":"ContainerDied","Data":"5f624f5a98ca55385b516aae4a3b9348c10970c64bdf39d0e22f4f55c240198a"}
Oct 30 03:28:31.805710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.805438    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"
Oct 30 03:28:31.805710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.805534    4877 scope.go:122] "RemoveContainer" containerID="7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1"
Oct 30 03:28:31.807832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.807800    4877 generic.go:356] "Generic (PLEG): container finished" podID="a9e55ebd-8a02-4595-bcc1-d3625030f099" containerID="7dd19f1e37efe2737f5e909bf1525b3c12e5fc9cbf46a40a81f4ed20545d2cda" exitCode=0
Oct 30 03:28:31.808066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.808022    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh" event={"ID":"a9e55ebd-8a02-4595-bcc1-d3625030f099","Type":"ContainerDied","Data":"7dd19f1e37efe2737f5e909bf1525b3c12e5fc9cbf46a40a81f4ed20545d2cda"}
Oct 30 03:28:31.809908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.809883    4877 generic.go:356] "Generic (PLEG): container finished" podID="ada11452-b43e-4475-ab9e-321aa368f965" containerID="21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29" exitCode=0
Oct 30 03:28:31.809987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.809961    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" event={"ID":"ada11452-b43e-4475-ab9e-321aa368f965","Type":"ContainerDied","Data":"21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29"}
Oct 30 03:28:31.810042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.810002    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l" event={"ID":"ada11452-b43e-4475-ab9e-321aa368f965","Type":"ContainerDied","Data":"f72f1feff32fb52dc9542edcafba9a92356ab64b6579aee6d8faa418bfff36d2"}
Oct 30 03:28:31.810076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.810069    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"
Oct 30 03:28:31.817771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.817755    4877 scope.go:122] "RemoveContainer" containerID="7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1"
Oct 30 03:28:31.818107 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:31.818085    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1\": container with ID starting with 7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1 not found: ID does not exist" containerID="7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1"
Oct 30 03:28:31.818217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.818112    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1"} err="failed to get container status \"7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1\": rpc error: code = NotFound desc = could not find container \"7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1\": container with ID starting with 7282d1bf574a39443ec367683acbe4ba2c4a73835f6c96af8365ee79225127f1 not found: ID does not exist"
Oct 30 03:28:31.818217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.818130    4877 scope.go:122] "RemoveContainer" containerID="21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29"
Oct 30 03:28:31.828509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.828486    4877 scope.go:122] "RemoveContainer" containerID="21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29"
Oct 30 03:28:31.828788 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:31.828763    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29\": container with ID starting with 21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29 not found: ID does not exist" containerID="21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29"
Oct 30 03:28:31.828859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.828799    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29"} err="failed to get container status \"21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29\": rpc error: code = NotFound desc = could not find container \"21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29\": container with ID starting with 21a0df9192122865a8329805b76b295b18daf4c394ae133ef638d44313ae8b29 not found: ID does not exist"
Oct 30 03:28:31.856747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.856637    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"]
Oct 30 03:28:31.860911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.860876    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c106c-56867c7b67-hdh9l"]
Oct 30 03:28:31.934715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.934637    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"]
Oct 30 03:28:31.940106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:31.940065    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-695c8457d4-jh9t5"]
Oct 30 03:28:33.035475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.035444    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:33.106446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.106340    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca\") pod \"a9e55ebd-8a02-4595-bcc1-d3625030f099\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") "
Oct 30 03:28:33.106446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.106390    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca\") pod \"a9e55ebd-8a02-4595-bcc1-d3625030f099\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") "
Oct 30 03:28:33.106708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.106507    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k\" (UniqueName: \"kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k\") pod \"a9e55ebd-8a02-4595-bcc1-d3625030f099\" (UID: \"a9e55ebd-8a02-4595-bcc1-d3625030f099\") "
Oct 30 03:28:33.107078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.107048    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca" pod "a9e55ebd-8a02-4595-bcc1-d3625030f099" (UID: "a9e55ebd-8a02-4595-bcc1-d3625030f099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:33.109395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.109360    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca" pod "a9e55ebd-8a02-4595-bcc1-d3625030f099" (UID: "a9e55ebd-8a02-4595-bcc1-d3625030f099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:33.109545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.109409    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k" pod "a9e55ebd-8a02-4595-bcc1-d3625030f099" (UID: "a9e55ebd-8a02-4595-bcc1-d3625030f099"). InnerVolumeSpecName "kube-api-access-n2j2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:33.207858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.207815    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2j2k\" (UniqueName: \"kubernetes.io/projected/a9e55ebd-8a02-4595-bcc1-d3625030f099-kube-api-access-n2j2k\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:33.207858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.207850    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:33.207858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.207863    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e55ebd-8a02-4595-bcc1-d3625030f099-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:33.743494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.743445    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" path="/var/lib/kubelet/pods/3ad05ccb-8d11-4718-826b-5d05bcabe7a1/volumes"
Oct 30 03:28:33.744093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.744065    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada11452-b43e-4475-ab9e-321aa368f965" path="/var/lib/kubelet/pods/ada11452-b43e-4475-ab9e-321aa368f965/volumes"
Oct 30 03:28:33.857130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.857094    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"
Oct 30 03:28:33.857130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.857118    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh" event={"ID":"a9e55ebd-8a02-4595-bcc1-d3625030f099","Type":"ContainerDied","Data":"d3c8ad500e6b3c547c58b524f20592b4ab625c9fb15d5a328129d7e4e30f5dc8"}
Oct 30 03:28:33.857343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:33.857187    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3c8ad500e6b3c547c58b524f20592b4ab625c9fb15d5a328129d7e4e30f5dc8"
Oct 30 03:28:34.986967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.986924    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"]
Oct 30 03:28:34.987834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.987813    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" containerName="payload-processing"
Oct 30 03:28:34.987924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.987836    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" containerName="payload-processing"
Oct 30 03:28:34.987924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.987858    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a9e55ebd-8a02-4595-bcc1-d3625030f099" containerName="revoke-keys"
Oct 30 03:28:34.987924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.987866    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e55ebd-8a02-4595-bcc1-d3625030f099" containerName="revoke-keys"
Oct 30 03:28:34.987924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.987884    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ada11452-b43e-4475-ab9e-321aa368f965" containerName="payload-pre-processing"
Oct 30 03:28:34.987924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.987893    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada11452-b43e-4475-ab9e-321aa368f965" containerName="payload-pre-processing"
Oct 30 03:28:34.988187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.988042    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad05ccb-8d11-4718-826b-5d05bcabe7a1" containerName="payload-processing"
Oct 30 03:28:34.988187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.988056    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e55ebd-8a02-4595-bcc1-d3625030f099" containerName="revoke-keys"
Oct 30 03:28:34.988187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.988066    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada11452-b43e-4475-ab9e-321aa368f965" containerName="payload-pre-processing"
Oct 30 03:28:34.993286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:34.993264    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.011235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.011205    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"]
Oct 30 03:28:35.017804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.017734    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:28:35.017804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.017779    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:35.141363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.141322    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlm7z\" (UniqueName: \"kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.141662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.141586    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.141662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.141631    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.243059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.242971    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.243059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.243020    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.243259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.243089    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm7z\" (UniqueName: \"kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.243754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.243732    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.245454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.245396    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.261941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.261910    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm7z\" (UniqueName: \"kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z\") pod \"maas-api-revoke-sub-ac57b6c6-pbwhl\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.311408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.311375    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:35.652000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.651968    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"]
Oct 30 03:28:35.652323 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:35.652301    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e9334_afb6_4bbc_81a2_21170fd676c5.slice/crio-69b48f66fa99813b9f1ee63bd7dcc0e0ede20762e9342e8e72c5960739ee171e WatchSource:0}: Error finding container 69b48f66fa99813b9f1ee63bd7dcc0e0ede20762e9342e8e72c5960739ee171e: Status 404 returned error can't find the container with id 69b48f66fa99813b9f1ee63bd7dcc0e0ede20762e9342e8e72c5960739ee171e
Oct 30 03:28:35.872217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:35.872175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl" event={"ID":"695e9334-afb6-4bbc-81a2-21170fd676c5","Type":"ContainerStarted","Data":"69b48f66fa99813b9f1ee63bd7dcc0e0ede20762e9342e8e72c5960739ee171e"}
Oct 30 03:28:36.882433 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:36.882378    4877 generic.go:356] "Generic (PLEG): container finished" podID="695e9334-afb6-4bbc-81a2-21170fd676c5" containerID="1a0cc079df00a6332556364fc9eb05a54d5392d8e130d0e74fde54fe48548998" exitCode=0
Oct 30 03:28:36.882900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:36.882457    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl" event={"ID":"695e9334-afb6-4bbc-81a2-21170fd676c5","Type":"ContainerDied","Data":"1a0cc079df00a6332556364fc9eb05a54d5392d8e130d0e74fde54fe48548998"}
Oct 30 03:28:38.044990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.044964    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:38.176295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.176211    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca\") pod \"695e9334-afb6-4bbc-81a2-21170fd676c5\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") "
Oct 30 03:28:38.176295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.176288    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca\") pod \"695e9334-afb6-4bbc-81a2-21170fd676c5\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") "
Oct 30 03:28:38.176558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.176395    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z\" (UniqueName: \"kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z\") pod \"695e9334-afb6-4bbc-81a2-21170fd676c5\" (UID: \"695e9334-afb6-4bbc-81a2-21170fd676c5\") "
Oct 30 03:28:38.176712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.176684    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca" pod "695e9334-afb6-4bbc-81a2-21170fd676c5" (UID: "695e9334-afb6-4bbc-81a2-21170fd676c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:38.176862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.176846    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:38.179073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.179044    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca" pod "695e9334-afb6-4bbc-81a2-21170fd676c5" (UID: "695e9334-afb6-4bbc-81a2-21170fd676c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:38.179192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.179092    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z" pod "695e9334-afb6-4bbc-81a2-21170fd676c5" (UID: "695e9334-afb6-4bbc-81a2-21170fd676c5"). InnerVolumeSpecName "kube-api-access-tlm7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:38.277631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.277594    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlm7z\" (UniqueName: \"kubernetes.io/projected/695e9334-afb6-4bbc-81a2-21170fd676c5-kube-api-access-tlm7z\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:38.277631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.277621    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695e9334-afb6-4bbc-81a2-21170fd676c5-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:38.898640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.898608    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"
Oct 30 03:28:38.898640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.898633    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl" event={"ID":"695e9334-afb6-4bbc-81a2-21170fd676c5","Type":"ContainerDied","Data":"69b48f66fa99813b9f1ee63bd7dcc0e0ede20762e9342e8e72c5960739ee171e"}
Oct 30 03:28:38.898909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:38.898683    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69b48f66fa99813b9f1ee63bd7dcc0e0ede20762e9342e8e72c5960739ee171e"
Oct 30 03:28:41.469279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:41.469240    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"]
Oct 30 03:28:41.469844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:41.469513    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy" containerID="cri-o://0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a" gracePeriod=30
Oct 30 03:28:42.632254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.632214    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"]
Oct 30 03:28:42.633045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.633026    4877 cpu_manager.go:414] "containerMap: removing container" podUID="695e9334-afb6-4bbc-81a2-21170fd676c5" containerName="revoke-keys"
Oct 30 03:28:42.633045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.633045    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695e9334-afb6-4bbc-81a2-21170fd676c5" containerName="revoke-keys"
Oct 30 03:28:42.633187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.633178    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="695e9334-afb6-4bbc-81a2-21170fd676c5" containerName="revoke-keys"
Oct 30 03:28:42.636693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.636675    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.653285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.653258    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"]
Oct 30 03:28:42.663543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.663516    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:28:42.663663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.663614    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:28:42.724963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.724918    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k748t\" (UniqueName: \"kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.725171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.724994    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.725171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.725030    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.825962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.825920    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k748t\" (UniqueName: \"kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.826160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.825976    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.826160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.826016    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.826740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.826716    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.828270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.828244    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.844172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.844150    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k748t\" (UniqueName: \"kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t\") pod \"maas-api-revoke-sub-dd15d502-v86tk\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:42.950216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:42.950140    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:43.258115 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:43.258080    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef827b08_faae_48a4_969e_e15a9f5101c0.slice/crio-e3efee5d0aac617d87a300f17ab2c31d8801f0870d60e0f0dac199825ef5d62c WatchSource:0}: Error finding container e3efee5d0aac617d87a300f17ab2c31d8801f0870d60e0f0dac199825ef5d62c: Status 404 returned error can't find the container with id e3efee5d0aac617d87a300f17ab2c31d8801f0870d60e0f0dac199825ef5d62c
Oct 30 03:28:43.258220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:43.258139    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"]
Oct 30 03:28:43.939206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:43.939109    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk" event={"ID":"ef827b08-faae-48a4-969e-e15a9f5101c0","Type":"ContainerStarted","Data":"7df43414f03f6bfe75a688e1db1ef8b8589182f4506404a77b0057cfae176117"}
Oct 30 03:28:43.939206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:43.939175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk" event={"ID":"ef827b08-faae-48a4-969e-e15a9f5101c0","Type":"ContainerStarted","Data":"e3efee5d0aac617d87a300f17ab2c31d8801f0870d60e0f0dac199825ef5d62c"}
Oct 30 03:28:43.977535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:43.977482    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk" podStartSLOduration=1.977469285 podStartE2EDuration="1.977469285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:43.975103086 +0000 UTC m=+1552.403992507" watchObservedRunningTime="2026-10-30 03:28:43.977469285 +0000 UTC m=+1552.406358703"
Oct 30 03:28:44.947870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:44.947832    4877 generic.go:356] "Generic (PLEG): container finished" podID="ef827b08-faae-48a4-969e-e15a9f5101c0" containerID="7df43414f03f6bfe75a688e1db1ef8b8589182f4506404a77b0057cfae176117" exitCode=0
Oct 30 03:28:44.948347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:44.947916    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk" event={"ID":"ef827b08-faae-48a4-969e-e15a9f5101c0","Type":"ContainerDied","Data":"7df43414f03f6bfe75a688e1db1ef8b8589182f4506404a77b0057cfae176117"}
Oct 30 03:28:46.149116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.149089    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:46.156890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.156859    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca\") pod \"ef827b08-faae-48a4-969e-e15a9f5101c0\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") "
Oct 30 03:28:46.156999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.156912    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t\" (UniqueName: \"kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t\") pod \"ef827b08-faae-48a4-969e-e15a9f5101c0\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") "
Oct 30 03:28:46.157047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.157025    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca\") pod \"ef827b08-faae-48a4-969e-e15a9f5101c0\" (UID: \"ef827b08-faae-48a4-969e-e15a9f5101c0\") "
Oct 30 03:28:46.157342 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.157316    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca" pod "ef827b08-faae-48a4-969e-e15a9f5101c0" (UID: "ef827b08-faae-48a4-969e-e15a9f5101c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:46.157566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.157548    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.158836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.158810    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca" pod "ef827b08-faae-48a4-969e-e15a9f5101c0" (UID: "ef827b08-faae-48a4-969e-e15a9f5101c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:46.158918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.158873    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t" pod "ef827b08-faae-48a4-969e-e15a9f5101c0" (UID: "ef827b08-faae-48a4-969e-e15a9f5101c0"). InnerVolumeSpecName "kube-api-access-k748t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:46.238134 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:46.238051    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:28:46.258284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.258253    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef827b08-faae-48a4-969e-e15a9f5101c0-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.258284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.258281    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k748t\" (UniqueName: \"kubernetes.io/projected/ef827b08-faae-48a4-969e-e15a9f5101c0-kube-api-access-k748t\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.748953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.748889    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:28:46.770354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770327    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770385    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770409    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770546    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770596    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770639    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770660    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:46.770841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770773    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770880    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:46.770991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770890    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.770991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.770965    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert\") pod \"72e18eab-f08a-4813-8cef-c13b0c627150\" (UID: \"72e18eab-f08a-4813-8cef-c13b0c627150\") "
Oct 30 03:28:46.771172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771143    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:46.771800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771373    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:46.771800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771595    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.771800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771618    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.771800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771633    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.771800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771648    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e18eab-f08a-4813-8cef-c13b0c627150-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.772109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.771996    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:46.773156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.773124    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:46.773156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.773144    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "kube-api-access-8cwfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:46.773418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.773392    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:28:46.773579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.773559    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy" pod "72e18eab-f08a-4813-8cef-c13b0c627150" (UID: "72e18eab-f08a-4813-8cef-c13b0c627150"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:46.872401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.872368    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.872401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.872399    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e18eab-f08a-4813-8cef-c13b0c627150-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.872401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.872409    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e18eab-f08a-4813-8cef-c13b0c627150-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.872680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.872417    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.872680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.872439    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwfg\" (UniqueName: \"kubernetes.io/projected/72e18eab-f08a-4813-8cef-c13b0c627150-kube-api-access-8cwfg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:28:46.965883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.965847    4877 generic.go:356] "Generic (PLEG): container finished" podID="72e18eab-f08a-4813-8cef-c13b0c627150" containerID="0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a" exitCode=0
Oct 30 03:28:46.966086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.965918    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"
Oct 30 03:28:46.966086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.965931    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" event={"ID":"72e18eab-f08a-4813-8cef-c13b0c627150","Type":"ContainerDied","Data":"0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a"}
Oct 30 03:28:46.966086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.965984    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc" event={"ID":"72e18eab-f08a-4813-8cef-c13b0c627150","Type":"ContainerDied","Data":"64649560bfc2fcfae3901beff8b32c4f316bfa1be6c5da0c71acc227cc5809c1"}
Oct 30 03:28:46.966086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.966014    4877 scope.go:122] "RemoveContainer" containerID="0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a"
Oct 30 03:28:46.967575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.967553    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"
Oct 30 03:28:46.967704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.967581    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk" event={"ID":"ef827b08-faae-48a4-969e-e15a9f5101c0","Type":"ContainerDied","Data":"e3efee5d0aac617d87a300f17ab2c31d8801f0870d60e0f0dac199825ef5d62c"}
Oct 30 03:28:46.967704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.967626    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3efee5d0aac617d87a300f17ab2c31d8801f0870d60e0f0dac199825ef5d62c"
Oct 30 03:28:46.978298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.978132    4877 scope.go:122] "RemoveContainer" containerID="0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a"
Oct 30 03:28:46.978696 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:28:46.978670    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a\": container with ID starting with 0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a not found: ID does not exist" containerID="0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a"
Oct 30 03:28:46.978803 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:46.978705    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a"} err="failed to get container status \"0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a\": rpc error: code = NotFound desc = could not find container \"0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a\": container with ID starting with 0af55cf280d84bda134b240ad0cef634e846f060f9bc495a69df76353312405a not found: ID does not exist"
Oct 30 03:28:47.019875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:47.019783    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"]
Oct 30 03:28:47.027437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:47.027396    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c106c-openshift-default-667dbc4649-twpcc"]
Oct 30 03:28:47.700313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:47.700277    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" path="/var/lib/kubelet/pods/72e18eab-f08a-4813-8cef-c13b0c627150/volumes"
Oct 30 03:28:54.518592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:54.518551    4877 scope.go:122] "RemoveContainer" containerID="5233cf5cdcc600b246f940c9b35f4351321bc27a5b999d4c2e78236ad7c3a761"
Oct 30 03:28:57.864337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.864302    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"]
Oct 30 03:28:57.864864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.864848    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ef827b08-faae-48a4-969e-e15a9f5101c0" containerName="revoke-keys"
Oct 30 03:28:57.864918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.864864    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef827b08-faae-48a4-969e-e15a9f5101c0" containerName="revoke-keys"
Oct 30 03:28:57.864918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.864895    4877 cpu_manager.go:414] "containerMap: removing container" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy"
Oct 30 03:28:57.864918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.864901    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy"
Oct 30 03:28:57.865016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.865001    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef827b08-faae-48a4-969e-e15a9f5101c0" containerName="revoke-keys"
Oct 30 03:28:57.865016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.865012    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e18eab-f08a-4813-8cef-c13b0c627150" containerName="istio-proxy"
Oct 30 03:28:57.870350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.870328    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.900930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.900899    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"]
Oct 30 03:28:57.918928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.918892    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd5pb\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.918943    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919059    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919108    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919173    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919262    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919290    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919306    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:57.919327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:57.919324    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.020982    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd5pb\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021047    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021109    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021150    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021193    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021291    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021334    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021359    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021392    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021573    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021615    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021690    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.021847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.021823    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.022333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.022307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.023512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.023490    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.023725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.023707    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.042160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.042129    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.042375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.042350    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd5pb\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.185459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.185362    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:58.510560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.510529    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"]
Oct 30 03:28:58.510712 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:28:58.510688    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77492b62_38b7_40d7_a7ef_1cda9ae8d778.slice/crio-97b15a9c6ae27cafa02a9ecf495caa30eb85834cc945ad1ef6d7d3b119c853bb WatchSource:0}: Error finding container 97b15a9c6ae27cafa02a9ecf495caa30eb85834cc945ad1ef6d7d3b119c853bb: Status 404 returned error can't find the container with id 97b15a9c6ae27cafa02a9ecf495caa30eb85834cc945ad1ef6d7d3b119c853bb
Oct 30 03:28:58.513155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.513124    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:58.513228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.513186    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:58.513228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:58.513215    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:28:59.080466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:59.076737    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" event={"ID":"77492b62-38b7-40d7-a7ef-1cda9ae8d778","Type":"ContainerStarted","Data":"3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25"}
Oct 30 03:28:59.080466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:59.076807    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" event={"ID":"77492b62-38b7-40d7-a7ef-1cda9ae8d778","Type":"ContainerStarted","Data":"97b15a9c6ae27cafa02a9ecf495caa30eb85834cc945ad1ef6d7d3b119c853bb"}
Oct 30 03:28:59.136211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:59.136140    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podStartSLOduration=2.136122119 podStartE2EDuration="2.136122119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:59.133166556 +0000 UTC m=+1567.562055975" watchObservedRunningTime="2026-10-30 03:28:59.136122119 +0000 UTC m=+1567.565011539"
Oct 30 03:28:59.188014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:59.187934    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:28:59.190090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:59.190060    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 03:28:59.190368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:28:59.190341    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 03:29:00.198446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:00.193192    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 03:29:00.198446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:00.193248    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 03:29:01.192447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:01.191844    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 03:29:01.192447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:01.191911    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 03:29:02.196449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:02.188828    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 03:29:02.196449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:02.188903    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 03:29:03.187450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:03.185876    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 03:29:03.187450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:03.185941    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 03:29:04.197516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:04.187869    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 03:29:04.197516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:04.187932    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 03:29:05.708941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.708910    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:29:05.709836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.709814    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"]
Oct 30 03:29:05.722517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.722490    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.723406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.722379    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:29:05.723760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.723745    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:29:05.738481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.738396    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-65f21d-openshift-default-dockercfg-b7p8p\""
Oct 30 03:29:05.738715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.738694    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"]
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852353    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852436    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9j8g\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852475    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852529    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852610    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852635    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852675    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852774    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.856052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.852988    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.954354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.954314    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.954715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.954698    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.954877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.954864    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.954987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.954973    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9j8g\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955045    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955075    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955115    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955162    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955186    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955215    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.955311    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.956023    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.956459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.956398    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.957164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.957139    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.959080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.959018    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.959226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.959207    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.974981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.974951    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9j8g\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:05.974981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:05.974967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token\") pod \"e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") " pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:06.049635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:06.049596    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:06.418146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:06.418113    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"]
Oct 30 03:29:06.418919 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:29:06.418891    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea18046_9d39_481e_bb13_bd5a848f0a23.slice/crio-293ee12ac837ea7c3aacebc2932bdb041ec55e6680f04329da40bff9423aaa72 WatchSource:0}: Error finding container 293ee12ac837ea7c3aacebc2932bdb041ec55e6680f04329da40bff9423aaa72: Status 404 returned error can't find the container with id 293ee12ac837ea7c3aacebc2932bdb041ec55e6680f04329da40bff9423aaa72
Oct 30 03:29:06.421669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:06.421639    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:06.421776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:06.421715    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:06.421776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:06.421762    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:07.225446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:07.224270    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" event={"ID":"fea18046-9d39-481e-bb13-bd5a848f0a23","Type":"ContainerStarted","Data":"5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"}
Oct 30 03:29:07.225446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:07.224337    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" event={"ID":"fea18046-9d39-481e-bb13-bd5a848f0a23","Type":"ContainerStarted","Data":"293ee12ac837ea7c3aacebc2932bdb041ec55e6680f04329da40bff9423aaa72"}
Oct 30 03:29:07.272541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:07.270974    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podStartSLOduration=2.27095523 podStartE2EDuration="2.27095523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:07.267207043 +0000 UTC m=+1575.696096486" watchObservedRunningTime="2026-10-30 03:29:07.27095523 +0000 UTC m=+1575.699844648"
Oct 30 03:29:08.050497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:08.050459    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:08.056046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:08.056021    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:08.233170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:08.233132    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:08.239652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:08.239621    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:11.608448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.607453    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"]
Oct 30 03:29:11.614445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.612800    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.631488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.630561    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"]
Oct 30 03:29:11.636911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.636072    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-65f21d\""
Oct 30 03:29:11.636911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.636678    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-65f21d-dockercfg-rtsgh\""
Oct 30 03:29:11.705411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.705378    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnqk\" (UniqueName: \"kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk\") pod \"payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.705581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.705433    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.806710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.806665    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnqk\" (UniqueName: \"kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk\") pod \"payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.806923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.806719    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.807644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.807621    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.826192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.826164    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnqk\" (UniqueName: \"kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk\") pod \"payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:11.946886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:11.946797    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:12.265834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:12.265811    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"]
Oct 30 03:29:12.269145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:12.269110    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v" event={"ID":"3d5e6c09-a67a-476c-846a-12beb2307cac","Type":"ContainerStarted","Data":"6de76a98f8825e35703b6b9d655bc4c0df23d894ba0c79e670f5420ab52b1e75"}
Oct 30 03:29:12.922185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:12.922143    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"]
Oct 30 03:29:12.926472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:12.926447    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:12.959096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:12.959061    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"]
Oct 30 03:29:13.023643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.023602    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.023792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.023659    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.023853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.023824    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.023892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.023866    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.023960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.023943    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzxm\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.024001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.023979    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.024137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.024120    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.024179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.024167    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.024242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.024226    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.125859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.125819    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.125859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.125864    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.125939    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.125964    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.125997    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzxm\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126028    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126155    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126214    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126245    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126301    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126353    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126441    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.126771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.126746    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.127088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.127066    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.129110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.129087    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.129280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.129264    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.145307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.145287    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzxm\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.146492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.146467    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.242684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.242603    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:13.279992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.279713    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v" event={"ID":"3d5e6c09-a67a-476c-846a-12beb2307cac","Type":"ContainerStarted","Data":"0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb"}
Oct 30 03:29:13.279992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.279806    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:13.322512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.322408    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v" podStartSLOduration=2.322390367 podStartE2EDuration="2.322390367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:13.321015913 +0000 UTC m=+1581.749905332" watchObservedRunningTime="2026-10-30 03:29:13.322390367 +0000 UTC m=+1581.751279786"
Oct 30 03:29:13.569930 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:29:13.569895    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0e9eaa_3db7_4cd1_91dd_0653ce9f4a8c.slice/crio-c54ddd04c47e2d5920518d4fcdd0cc63697a7f4e0487eee88808ed38cd69f2ca WatchSource:0}: Error finding container c54ddd04c47e2d5920518d4fcdd0cc63697a7f4e0487eee88808ed38cd69f2ca: Status 404 returned error can't find the container with id c54ddd04c47e2d5920518d4fcdd0cc63697a7f4e0487eee88808ed38cd69f2ca
Oct 30 03:29:13.570759 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.570732    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"]
Oct 30 03:29:13.572667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.572639    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:13.572775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.572716    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:13.572775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:13.572742    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:14.307447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:14.307396    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" event={"ID":"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c","Type":"ContainerStarted","Data":"b5cbe3c6bf179a3270c5f136f8e4796eba4e10b7de9121b505f451a93b96bfc8"}
Oct 30 03:29:14.308444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:14.308129    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" event={"ID":"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c","Type":"ContainerStarted","Data":"c54ddd04c47e2d5920518d4fcdd0cc63697a7f4e0487eee88808ed38cd69f2ca"}
Oct 30 03:29:14.350894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:14.350842    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podStartSLOduration=2.350827597 podStartE2EDuration="2.350827597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:14.347166881 +0000 UTC m=+1582.776056311" watchObservedRunningTime="2026-10-30 03:29:14.350827597 +0000 UTC m=+1582.779717072"
Oct 30 03:29:15.243409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:15.243374    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:15.245145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:15.245120    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:29:15.245262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:15.245171    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 03:29:16.229140 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:29:16.229100    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:29:16.243746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.243714    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:29:16.243901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.243785    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 03:29:16.304734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.304223    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"]
Oct 30 03:29:16.315442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.315362    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.318437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.318393    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"]
Oct 30 03:29:16.473668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.473629    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.473668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.473671    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.473878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.473782    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.473878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.473838    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.473954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.473931    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.474000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.473961    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.474099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.474075    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.474170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.474148    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtht\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.474318 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.474300    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.575807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.575772    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.575807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.575809    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.575861    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.575894    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.575958    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.575992    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtht\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.576089    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.576137    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.576170    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.576290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.576227    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.595446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.576934    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.595446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.581446    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.595446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.581827    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.595446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.581951    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.599447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.596933    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.608151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.608014    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.608992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.608964    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.610462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.609819    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtht\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:16.635949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:16.635928    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:17.005155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.005127    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"]
Oct 30 03:29:17.005907 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:29:17.005881    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3a22e1_6bcf_405d_b714_d4e6d131e148.slice/crio-cd1e0b6aee46b86bb102cead66d767c40a278330bef64443aac37c147f60ef6f WatchSource:0}: Error finding container cd1e0b6aee46b86bb102cead66d767c40a278330bef64443aac37c147f60ef6f: Status 404 returned error can't find the container with id cd1e0b6aee46b86bb102cead66d767c40a278330bef64443aac37c147f60ef6f
Oct 30 03:29:17.008633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.008602    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:17.008727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.008664    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:17.008727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.008692    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:17.243823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.243789    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:29:17.244441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.244390    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 03:29:17.339601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.339559    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" event={"ID":"ae3a22e1-6bcf-405d-b714-d4e6d131e148","Type":"ContainerStarted","Data":"0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60"}
Oct 30 03:29:17.339788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.339628    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" event={"ID":"ae3a22e1-6bcf-405d-b714-d4e6d131e148","Type":"ContainerStarted","Data":"cd1e0b6aee46b86bb102cead66d767c40a278330bef64443aac37c147f60ef6f"}
Oct 30 03:29:17.404446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.399525    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podStartSLOduration=1.39950355 podStartE2EDuration="1.39950355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:17.394383031 +0000 UTC m=+1585.823272473" watchObservedRunningTime="2026-10-30 03:29:17.39950355 +0000 UTC m=+1585.828392968"
Oct 30 03:29:17.638549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:17.637525    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:18.243203 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:18.243171    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:29:18.243374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:18.243223    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 03:29:18.639469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:18.639239    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:29:18.639469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:18.639302    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:29:19.243843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:19.243796    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:29:19.244036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:19.243864    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 03:29:19.637003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:19.636877    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:29:19.637003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:19.636954    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:29:20.243269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:20.243234    4877 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 30 03:29:20.243840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:20.243286    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 30 03:29:20.639447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:20.636994    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:29:20.639447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:20.637075    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:29:21.248699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:21.248664    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:21.249176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:21.249014    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:21.249859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:21.249836    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:29:21.636530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:21.636489    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:29:21.636742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:21.636555    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:29:22.636623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:22.636586    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:29:22.637134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:22.636648    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:29:23.636808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:23.636771    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 03:29:23.637229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:23.636836    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 03:29:24.310376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:24.310344    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:29:24.640828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:24.640753    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:24.641225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:24.641210    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:24.641974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:24.641948    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:29:45.637247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:45.637204    4877 generic.go:356] "Generic (PLEG): container finished" podID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerID="5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863" exitCode=0
Oct 30 03:29:45.637717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:45.637278    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" event={"ID":"fea18046-9d39-481e-bb13-bd5a848f0a23","Type":"ContainerDied","Data":"5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"}
Oct 30 03:29:45.637768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:45.637734    4877 scope.go:122] "RemoveContainer" containerID="5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"
Oct 30 03:29:45.639238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:45.639208    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:45.639327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:45.639302    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:45.639368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:45.639334    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:46.234512 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:29:46.234472    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:29:46.611304 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.611267    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"]
Oct 30 03:29:46.618730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.618700    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.635254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.634938    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-773fbb-openshift-default-dockercfg-wf8qd\""
Oct 30 03:29:46.653327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.653287    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" event={"ID":"fea18046-9d39-481e-bb13-bd5a848f0a23","Type":"ContainerStarted","Data":"d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a"}
Oct 30 03:29:46.655101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.655072    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"]
Oct 30 03:29:46.757212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757178    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757254    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgh8\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757362    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757393    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757445    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757538    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757571    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757588    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.757807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.757779    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860165    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860291    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860387    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgh8\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860517    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860548    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860609    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860659    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860725    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.860781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.860755    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.864483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.861874    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.864483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.863195    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.881447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.870378    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.901826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.901776    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.920442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.902487    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.920731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.902957    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgh8\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.921353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.921327    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.923453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.922928    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.936018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.935973    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo\") pod \"e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") " pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:46.938329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:46.937113    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:47.050923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.049830    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:47.050923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.050098    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 03:29:47.050923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.050146    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 03:29:47.305636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.305585    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"]
Oct 30 03:29:47.308175 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:29:47.308148    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a2a436_1bdd_48d4_b669_3eb5051cb190.slice/crio-ce7f97be264101e295b0589bd8e210f1b54f54fc962f1abbd56f10eba69196af WatchSource:0}: Error finding container ce7f97be264101e295b0589bd8e210f1b54f54fc962f1abbd56f10eba69196af: Status 404 returned error can't find the container with id ce7f97be264101e295b0589bd8e210f1b54f54fc962f1abbd56f10eba69196af
Oct 30 03:29:47.311071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.311026    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:47.311145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.311126    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:47.311196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.311180    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:29:47.664111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.664000    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" event={"ID":"23a2a436-1bdd-48d4-b669-3eb5051cb190","Type":"ContainerStarted","Data":"e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"}
Oct 30 03:29:47.664111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.664079    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" event={"ID":"23a2a436-1bdd-48d4-b669-3eb5051cb190","Type":"ContainerStarted","Data":"ce7f97be264101e295b0589bd8e210f1b54f54fc962f1abbd56f10eba69196af"}
Oct 30 03:29:47.720720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.720398    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podStartSLOduration=1.720380675 podStartE2EDuration="1.720380675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:47.719607081 +0000 UTC m=+1616.148496529" watchObservedRunningTime="2026-10-30 03:29:47.720380675 +0000 UTC m=+1616.149270094"
Oct 30 03:29:47.939065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:47.938979    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:48.051005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:48.050962    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 03:29:48.051220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:48.051028    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 03:29:48.944389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:48.944351    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:49.050511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:49.050478    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 03:29:49.050690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:49.050542    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 03:29:49.685449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:49.683396    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:49.715858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:49.715823    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:29:50.050063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:50.050027    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 03:29:50.050590 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:50.050089    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 03:29:51.050634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:51.050600    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 03:29:51.051030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:51.050656    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 03:29:52.156447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:52.155786    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:52.157336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:52.157312    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:29:52.159006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:29:52.158982    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:30:00.170600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.170549    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"]
Oct 30 03:30:00.179141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.178994    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"]
Oct 30 03:30:00.179793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.179768    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.184332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.183812    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"]
Oct 30 03:30:00.184501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.184368    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.188325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.188306    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.189080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.189063    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"]
Oct 30 03:30:00.205136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.204727    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"]
Oct 30 03:30:00.206139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.205983    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:30:00.206794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.206776    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.226230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.226094    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"]
Oct 30 03:30:00.226230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.226153    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"]
Oct 30 03:30:00.226230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.226197    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"]
Oct 30 03:30:00.226230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.226225    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"]
Oct 30 03:30:00.226546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.226329    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.241345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.241311    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"]
Oct 30 03:30:00.293768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.293736    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.293931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.293792    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.293931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.293910    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.294015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.293969    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npb9r\" (UniqueName: \"kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.294051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.294024    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk4c\" (UniqueName: \"kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c\") pod \"maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.294088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.294080    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7hz\" (UniqueName: \"kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.294163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.294148    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.294268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.294254    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmmw\" (UniqueName: \"kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.294317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.294300    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhsv\" (UniqueName: \"kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.294354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.294340    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.395596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395562    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk4c\" (UniqueName: \"kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c\") pod \"maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.395786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395640    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7hz\" (UniqueName: \"kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.395786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395673    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.395786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395774    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmmw\" (UniqueName: \"kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.395907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395818    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhsv\" (UniqueName: \"kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.395907 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395858    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.395997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395935    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.395997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.395976    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.396088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396006    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.396088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396029    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npb9r\" (UniqueName: \"kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.396611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396583    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.396760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396662    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.396760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.396760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396693    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.396890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.396758    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.415367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.415340    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhsv\" (UniqueName: \"kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.415498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.415343    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk4c\" (UniqueName: \"kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c\") pod \"maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.415498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.415470    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmmw\" (UniqueName: \"kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.415601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.415565    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7hz\" (UniqueName: \"kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.416131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.416108    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npb9r\" (UniqueName: \"kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.506370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.506290    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:00.528458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.528394    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:00.541652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.541619    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:00.568882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.568684    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:00.582346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.580643    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:00.830661 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:30:00.830628    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7dc6f0_e183_4c06_ba47_f0c8073a15ce.slice/crio-079fcca9220e10bd688a1e0d4ae91cad5f2d356b31dd7fdb1c6c66ad371fef83 WatchSource:0}: Error finding container 079fcca9220e10bd688a1e0d4ae91cad5f2d356b31dd7fdb1c6c66ad371fef83: Status 404 returned error can't find the container with id 079fcca9220e10bd688a1e0d4ae91cad5f2d356b31dd7fdb1c6c66ad371fef83
Oct 30 03:30:00.832580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.832551    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"]
Oct 30 03:30:00.858112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.858087    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"]
Oct 30 03:30:00.864735 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:30:00.864710    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31b95ae_2c3e_4a99_bdd1_8b13546cca10.slice/crio-fad3eeb39833e6e31500f4d7cb1ae798760544c0c809eb22848feeb97eb6ec9e WatchSource:0}: Error finding container fad3eeb39833e6e31500f4d7cb1ae798760544c0c809eb22848feeb97eb6ec9e: Status 404 returned error can't find the container with id fad3eeb39833e6e31500f4d7cb1ae798760544c0c809eb22848feeb97eb6ec9e
Oct 30 03:30:00.883655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.883635    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"]
Oct 30 03:30:00.948184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.948150    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"]
Oct 30 03:30:00.959142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:00.959118    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"]
Oct 30 03:30:00.979221 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:30:00.979164    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe83d78_b370_41f6_8ae0_d793d3fd3109.slice/crio-733b41d8e64c28177c002c6180752da01f291a5fa157335b746088bc3db94dba WatchSource:0}: Error finding container 733b41d8e64c28177c002c6180752da01f291a5fa157335b746088bc3db94dba: Status 404 returned error can't find the container with id 733b41d8e64c28177c002c6180752da01f291a5fa157335b746088bc3db94dba
Oct 30 03:30:00.980498 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:30:00.980452    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda969a7a1_7988_4819_9a9f_9caecf42b592.slice/crio-7402b52393f4fc88dfc6b9078be9081a7f39794fbaf0646595893a6b19d9c309 WatchSource:0}: Error finding container 7402b52393f4fc88dfc6b9078be9081a7f39794fbaf0646595893a6b19d9c309: Status 404 returned error can't find the container with id 7402b52393f4fc88dfc6b9078be9081a7f39794fbaf0646595893a6b19d9c309
Oct 30 03:30:01.817940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.817894    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerStarted","Data":"354bb39a5823e6435bddb6b862d5ae608cd3dec27108ea7d0a0863ee1450d577"}
Oct 30 03:30:01.818393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.817964    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerStarted","Data":"079fcca9220e10bd688a1e0d4ae91cad5f2d356b31dd7fdb1c6c66ad371fef83"}
Oct 30 03:30:01.820265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.820228    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" event={"ID":"a969a7a1-7988-4819-9a9f-9caecf42b592","Type":"ContainerStarted","Data":"e0ed0f9511cfea451e23644978c7571dd13c77e64107a3ac5da919acfc05c989"}
Oct 30 03:30:01.820399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.820296    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" event={"ID":"a969a7a1-7988-4819-9a9f-9caecf42b592","Type":"ContainerStarted","Data":"7402b52393f4fc88dfc6b9078be9081a7f39794fbaf0646595893a6b19d9c309"}
Oct 30 03:30:01.822014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.821987    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" event={"ID":"fbe83d78-b370-41f6-8ae0-d793d3fd3109","Type":"ContainerStarted","Data":"4914a5b7ca19ec75e496cd63eb0e7b22fa7c68cd1af2e69fb763e270fba95a13"}
Oct 30 03:30:01.822107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.822034    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" event={"ID":"fbe83d78-b370-41f6-8ae0-d793d3fd3109","Type":"ContainerStarted","Data":"733b41d8e64c28177c002c6180752da01f291a5fa157335b746088bc3db94dba"}
Oct 30 03:30:01.823863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.823816    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" event={"ID":"68216927-1902-41ec-a9ed-44264d7e064d","Type":"ContainerStarted","Data":"14e61f7250d8e499fc861195cfb47eedba430976fcf2c52ea55b6dff16f717cf"}
Oct 30 03:30:01.823863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.823860    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" event={"ID":"68216927-1902-41ec-a9ed-44264d7e064d","Type":"ContainerStarted","Data":"5e743f4c142b12f6145b8f12f28ebec7558cf0431a1906fb58488a95228120df"}
Oct 30 03:30:01.825651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.825616    4877 generic.go:356] "Generic (PLEG): container finished" podID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerID="a4614939c6c97cba1733b46c3c35f145205c5ad6cddf51f86b78c6f8d175037d" exitCode=60
Oct 30 03:30:01.825816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.825786    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" event={"ID":"a31b95ae-2c3e-4a99-bdd1-8b13546cca10","Type":"ContainerDied","Data":"a4614939c6c97cba1733b46c3c35f145205c5ad6cddf51f86b78c6f8d175037d"}
Oct 30 03:30:01.825926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.825847    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" event={"ID":"a31b95ae-2c3e-4a99-bdd1-8b13546cca10","Type":"ContainerStarted","Data":"fad3eeb39833e6e31500f4d7cb1ae798760544c0c809eb22848feeb97eb6ec9e"}
Oct 30 03:30:01.825926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.825874    4877 scope.go:122] "RemoveContainer" containerID="a4614939c6c97cba1733b46c3c35f145205c5ad6cddf51f86b78c6f8d175037d"
Oct 30 03:30:01.857464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.856017    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" podStartSLOduration=1.855997784 podStartE2EDuration="1.855997784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.853537139 +0000 UTC m=+1630.282426582" watchObservedRunningTime="2026-10-30 03:30:01.855997784 +0000 UTC m=+1630.284887204"
Oct 30 03:30:01.893180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.893136    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" podStartSLOduration=1.893121928 podStartE2EDuration="1.893121928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.890680378 +0000 UTC m=+1630.319569795" watchObservedRunningTime="2026-10-30 03:30:01.893121928 +0000 UTC m=+1630.322011346"
Oct 30 03:30:01.928959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.928911    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" podStartSLOduration=1.928896633 podStartE2EDuration="1.928896633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.925602255 +0000 UTC m=+1630.354491673" watchObservedRunningTime="2026-10-30 03:30:01.928896633 +0000 UTC m=+1630.357786082"
Oct 30 03:30:01.999775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:01.999725    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" podStartSLOduration=1.999712326 podStartE2EDuration="1.999712326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:01.997665297 +0000 UTC m=+1630.426554714" watchObservedRunningTime="2026-10-30 03:30:01.999712326 +0000 UTC m=+1630.428601743"
Oct 30 03:30:02.836008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.835970    4877 generic.go:356] "Generic (PLEG): container finished" podID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerID="e0ed0f9511cfea451e23644978c7571dd13c77e64107a3ac5da919acfc05c989" exitCode=60
Oct 30 03:30:02.836469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.836048    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" event={"ID":"a969a7a1-7988-4819-9a9f-9caecf42b592","Type":"ContainerDied","Data":"e0ed0f9511cfea451e23644978c7571dd13c77e64107a3ac5da919acfc05c989"}
Oct 30 03:30:02.836469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.836316    4877 scope.go:122] "RemoveContainer" containerID="e0ed0f9511cfea451e23644978c7571dd13c77e64107a3ac5da919acfc05c989"
Oct 30 03:30:02.837863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.837837    4877 generic.go:356] "Generic (PLEG): container finished" podID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerID="4914a5b7ca19ec75e496cd63eb0e7b22fa7c68cd1af2e69fb763e270fba95a13" exitCode=60
Oct 30 03:30:02.837971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.837914    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" event={"ID":"fbe83d78-b370-41f6-8ae0-d793d3fd3109","Type":"ContainerDied","Data":"4914a5b7ca19ec75e496cd63eb0e7b22fa7c68cd1af2e69fb763e270fba95a13"}
Oct 30 03:30:02.838183 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.838165    4877 scope.go:122] "RemoveContainer" containerID="4914a5b7ca19ec75e496cd63eb0e7b22fa7c68cd1af2e69fb763e270fba95a13"
Oct 30 03:30:02.840761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.840739    4877 generic.go:356] "Generic (PLEG): container finished" podID="68216927-1902-41ec-a9ed-44264d7e064d" containerID="14e61f7250d8e499fc861195cfb47eedba430976fcf2c52ea55b6dff16f717cf" exitCode=60
Oct 30 03:30:02.840858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.840808    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" event={"ID":"68216927-1902-41ec-a9ed-44264d7e064d","Type":"ContainerDied","Data":"14e61f7250d8e499fc861195cfb47eedba430976fcf2c52ea55b6dff16f717cf"}
Oct 30 03:30:02.841016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.840999    4877 scope.go:122] "RemoveContainer" containerID="14e61f7250d8e499fc861195cfb47eedba430976fcf2c52ea55b6dff16f717cf"
Oct 30 03:30:02.842765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.842744    4877 generic.go:356] "Generic (PLEG): container finished" podID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerID="b65a11a97de016b2db4396e3b35c8f37a991690452ffca101b2455ab195a42c1" exitCode=60
Oct 30 03:30:02.842856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.842819    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" event={"ID":"a31b95ae-2c3e-4a99-bdd1-8b13546cca10","Type":"ContainerDied","Data":"b65a11a97de016b2db4396e3b35c8f37a991690452ffca101b2455ab195a42c1"}
Oct 30 03:30:02.842916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.842886    4877 scope.go:122] "RemoveContainer" containerID="a4614939c6c97cba1733b46c3c35f145205c5ad6cddf51f86b78c6f8d175037d"
Oct 30 03:30:02.843052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.843034    4877 scope.go:122] "RemoveContainer" containerID="b65a11a97de016b2db4396e3b35c8f37a991690452ffca101b2455ab195a42c1"
Oct 30 03:30:02.843322 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:02.843289    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx_odh-ai-gateway-infra(a31b95ae-2c3e-4a99-bdd1-8b13546cca10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10"
Oct 30 03:30:02.844295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.844277    4877 generic.go:356] "Generic (PLEG): container finished" podID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerID="354bb39a5823e6435bddb6b862d5ae608cd3dec27108ea7d0a0863ee1450d577" exitCode=60
Oct 30 03:30:02.844372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.844298    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerDied","Data":"354bb39a5823e6435bddb6b862d5ae608cd3dec27108ea7d0a0863ee1450d577"}
Oct 30 03:30:02.844588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:02.844572    4877 scope.go:122] "RemoveContainer" containerID="354bb39a5823e6435bddb6b862d5ae608cd3dec27108ea7d0a0863ee1450d577"
Oct 30 03:30:03.854530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.854493    4877 generic.go:356] "Generic (PLEG): container finished" podID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerID="e814743079daab31ed50f40cf47134d29f596753dc19a909e0164bbdeca924fc" exitCode=60
Oct 30 03:30:03.855012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.854539    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" event={"ID":"a969a7a1-7988-4819-9a9f-9caecf42b592","Type":"ContainerDied","Data":"e814743079daab31ed50f40cf47134d29f596753dc19a909e0164bbdeca924fc"}
Oct 30 03:30:03.855012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.854597    4877 scope.go:122] "RemoveContainer" containerID="e0ed0f9511cfea451e23644978c7571dd13c77e64107a3ac5da919acfc05c989"
Oct 30 03:30:03.855012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.854917    4877 scope.go:122] "RemoveContainer" containerID="e814743079daab31ed50f40cf47134d29f596753dc19a909e0164bbdeca924fc"
Oct 30 03:30:03.855263 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:03.855239    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt_odh-ai-gateway-infra(a969a7a1-7988-4819-9a9f-9caecf42b592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592"
Oct 30 03:30:03.856692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.856675    4877 generic.go:356] "Generic (PLEG): container finished" podID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerID="f1728342959df76d26a85180c68f0a7808e43d35378a893c184661946e618ed4" exitCode=60
Oct 30 03:30:03.856802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.856762    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" event={"ID":"fbe83d78-b370-41f6-8ae0-d793d3fd3109","Type":"ContainerDied","Data":"f1728342959df76d26a85180c68f0a7808e43d35378a893c184661946e618ed4"}
Oct 30 03:30:03.857016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.856999    4877 scope.go:122] "RemoveContainer" containerID="f1728342959df76d26a85180c68f0a7808e43d35378a893c184661946e618ed4"
Oct 30 03:30:03.857282 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:03.857258    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68_odh-ai-gateway-infra(fbe83d78-b370-41f6-8ae0-d793d3fd3109)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109"
Oct 30 03:30:03.858743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.858716    4877 generic.go:356] "Generic (PLEG): container finished" podID="68216927-1902-41ec-a9ed-44264d7e064d" containerID="82fa0d0f4d16b592a4f10bd916d6c07643e2d71f613ef9a0dbe309b9aaee25c7" exitCode=60
Oct 30 03:30:03.858823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.858784    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" event={"ID":"68216927-1902-41ec-a9ed-44264d7e064d","Type":"ContainerDied","Data":"82fa0d0f4d16b592a4f10bd916d6c07643e2d71f613ef9a0dbe309b9aaee25c7"}
Oct 30 03:30:03.859036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.859022    4877 scope.go:122] "RemoveContainer" containerID="82fa0d0f4d16b592a4f10bd916d6c07643e2d71f613ef9a0dbe309b9aaee25c7"
Oct 30 03:30:03.859266 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:03.859244    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml_odh-ai-gateway-infra(68216927-1902-41ec-a9ed-44264d7e064d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" podUID="68216927-1902-41ec-a9ed-44264d7e064d"
Oct 30 03:30:03.860702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.860684    4877 scope.go:122] "RemoveContainer" containerID="b65a11a97de016b2db4396e3b35c8f37a991690452ffca101b2455ab195a42c1"
Oct 30 03:30:03.860925 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:03.860903    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx_odh-ai-gateway-infra(a31b95ae-2c3e-4a99-bdd1-8b13546cca10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10"
Oct 30 03:30:03.862640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:03.862617    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerStarted","Data":"f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36"}
Oct 30 03:30:04.012040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.012016    4877 scope.go:122] "RemoveContainer" containerID="4914a5b7ca19ec75e496cd63eb0e7b22fa7c68cd1af2e69fb763e270fba95a13"
Oct 30 03:30:04.025730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.025710    4877 scope.go:122] "RemoveContainer" containerID="14e61f7250d8e499fc861195cfb47eedba430976fcf2c52ea55b6dff16f717cf"
Oct 30 03:30:04.872290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.872259    4877 scope.go:122] "RemoveContainer" containerID="f1728342959df76d26a85180c68f0a7808e43d35378a893c184661946e618ed4"
Oct 30 03:30:04.872770 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:04.872579    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68_odh-ai-gateway-infra(fbe83d78-b370-41f6-8ae0-d793d3fd3109)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109"
Oct 30 03:30:04.874120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.874099    4877 scope.go:122] "RemoveContainer" containerID="82fa0d0f4d16b592a4f10bd916d6c07643e2d71f613ef9a0dbe309b9aaee25c7"
Oct 30 03:30:04.874378 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:04.874355    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml_odh-ai-gateway-infra(68216927-1902-41ec-a9ed-44264d7e064d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" podUID="68216927-1902-41ec-a9ed-44264d7e064d"
Oct 30 03:30:04.875747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.875726    4877 generic.go:356] "Generic (PLEG): container finished" podID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerID="f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36" exitCode=60
Oct 30 03:30:04.875824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.875804    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerDied","Data":"f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36"}
Oct 30 03:30:04.875876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.875866    4877 scope.go:122] "RemoveContainer" containerID="354bb39a5823e6435bddb6b862d5ae608cd3dec27108ea7d0a0863ee1450d577"
Oct 30 03:30:04.876021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.875999    4877 scope.go:122] "RemoveContainer" containerID="f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36"
Oct 30 03:30:04.876282 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:04.876255    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl_odh-ai-gateway-infra(7a7dc6f0-e183-4c06-ba47-f0c8073a15ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce"
Oct 30 03:30:04.877764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:04.877748    4877 scope.go:122] "RemoveContainer" containerID="e814743079daab31ed50f40cf47134d29f596753dc19a909e0164bbdeca924fc"
Oct 30 03:30:04.877953 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:04.877938    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt_odh-ai-gateway-infra(a969a7a1-7988-4819-9a9f-9caecf42b592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592"
Oct 30 03:30:05.886494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:05.886463    4877 scope.go:122] "RemoveContainer" containerID="f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36"
Oct 30 03:30:05.886906 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:05.886777    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl_odh-ai-gateway-infra(7a7dc6f0-e183-4c06-ba47-f0c8073a15ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce"
Oct 30 03:30:14.701444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:14.694962    4877 scope.go:122] "RemoveContainer" containerID="b65a11a97de016b2db4396e3b35c8f37a991690452ffca101b2455ab195a42c1"
Oct 30 03:30:14.701444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:14.695922    4877 scope.go:122] "RemoveContainer" containerID="82fa0d0f4d16b592a4f10bd916d6c07643e2d71f613ef9a0dbe309b9aaee25c7"
Oct 30 03:30:15.722269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.722234    4877 scope.go:122] "RemoveContainer" containerID="e814743079daab31ed50f40cf47134d29f596753dc19a909e0164bbdeca924fc"
Oct 30 03:30:15.972872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.972777    4877 generic.go:356] "Generic (PLEG): container finished" podID="68216927-1902-41ec-a9ed-44264d7e064d" containerID="47350dbdb082cb0fbe5e26460cbf8300d9fb54d777f5626c20e108869695d3c8" exitCode=60
Oct 30 03:30:15.972872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.972846    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" event={"ID":"68216927-1902-41ec-a9ed-44264d7e064d","Type":"ContainerDied","Data":"47350dbdb082cb0fbe5e26460cbf8300d9fb54d777f5626c20e108869695d3c8"}
Oct 30 03:30:15.973086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.972961    4877 scope.go:122] "RemoveContainer" containerID="82fa0d0f4d16b592a4f10bd916d6c07643e2d71f613ef9a0dbe309b9aaee25c7"
Oct 30 03:30:15.973451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.973408    4877 scope.go:122] "RemoveContainer" containerID="47350dbdb082cb0fbe5e26460cbf8300d9fb54d777f5626c20e108869695d3c8"
Oct 30 03:30:15.973827 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:15.973804    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml_odh-ai-gateway-infra(68216927-1902-41ec-a9ed-44264d7e064d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" podUID="68216927-1902-41ec-a9ed-44264d7e064d"
Oct 30 03:30:15.975169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.975149    4877 generic.go:356] "Generic (PLEG): container finished" podID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerID="e5a587d086b0b1213958c241af3357fc84264b0317112ca331e5da9849837710" exitCode=60
Oct 30 03:30:15.975252 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.975216    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" event={"ID":"a31b95ae-2c3e-4a99-bdd1-8b13546cca10","Type":"ContainerDied","Data":"e5a587d086b0b1213958c241af3357fc84264b0317112ca331e5da9849837710"}
Oct 30 03:30:15.975566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:15.975550    4877 scope.go:122] "RemoveContainer" containerID="e5a587d086b0b1213958c241af3357fc84264b0317112ca331e5da9849837710"
Oct 30 03:30:15.975835 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:15.975819    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx_odh-ai-gateway-infra(a31b95ae-2c3e-4a99-bdd1-8b13546cca10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10"
Oct 30 03:30:16.018542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:16.018518    4877 scope.go:122] "RemoveContainer" containerID="b65a11a97de016b2db4396e3b35c8f37a991690452ffca101b2455ab195a42c1"
Oct 30 03:30:16.237109 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:16.237035    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:30:17.023290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.023253    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"]
Oct 30 03:30:17.034413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.034385    4877 generic.go:356] "Generic (PLEG): container finished" podID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerID="772e13b50c4f4b8ba5fd625a62df8b60cf1d9cc31b5aa53ebbc49ff7d279cbc3" exitCode=60
Oct 30 03:30:17.034577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.034458    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" event={"ID":"a969a7a1-7988-4819-9a9f-9caecf42b592","Type":"ContainerDied","Data":"772e13b50c4f4b8ba5fd625a62df8b60cf1d9cc31b5aa53ebbc49ff7d279cbc3"}
Oct 30 03:30:17.034577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.034525    4877 scope.go:122] "RemoveContainer" containerID="e814743079daab31ed50f40cf47134d29f596753dc19a909e0164bbdeca924fc"
Oct 30 03:30:17.034958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.034936    4877 scope.go:122] "RemoveContainer" containerID="772e13b50c4f4b8ba5fd625a62df8b60cf1d9cc31b5aa53ebbc49ff7d279cbc3"
Oct 30 03:30:17.035307 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:17.035282    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt_odh-ai-gateway-infra(a969a7a1-7988-4819-9a9f-9caecf42b592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592"
Oct 30 03:30:17.065498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.065456    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"]
Oct 30 03:30:17.255368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.255025    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:17.350089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.350053    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca\") pod \"68216927-1902-41ec-a9ed-44264d7e064d\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") "
Oct 30 03:30:17.350361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.350339    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r\" (UniqueName: \"kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r\") pod \"68216927-1902-41ec-a9ed-44264d7e064d\" (UID: \"68216927-1902-41ec-a9ed-44264d7e064d\") "
Oct 30 03:30:17.351750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.351718    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca" pod "68216927-1902-41ec-a9ed-44264d7e064d" (UID: "68216927-1902-41ec-a9ed-44264d7e064d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:17.353469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.353404    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r" pod "68216927-1902-41ec-a9ed-44264d7e064d" (UID: "68216927-1902-41ec-a9ed-44264d7e064d"). InnerVolumeSpecName "kube-api-access-npb9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:17.421136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.421103    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:17.452007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.451980    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv\" (UniqueName: \"kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv\") pod \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") "
Oct 30 03:30:17.452165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.452101    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca\") pod \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\" (UID: \"a31b95ae-2c3e-4a99-bdd1-8b13546cca10\") "
Oct 30 03:30:17.452479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.452457    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca" pod "a31b95ae-2c3e-4a99-bdd1-8b13546cca10" (UID: "a31b95ae-2c3e-4a99-bdd1-8b13546cca10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:17.452479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.452474    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npb9r\" (UniqueName: \"kubernetes.io/projected/68216927-1902-41ec-a9ed-44264d7e064d-kube-api-access-npb9r\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.452570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.452488    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68216927-1902-41ec-a9ed-44264d7e064d-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.454121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.454099    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv" pod "a31b95ae-2c3e-4a99-bdd1-8b13546cca10" (UID: "a31b95ae-2c3e-4a99-bdd1-8b13546cca10"). InnerVolumeSpecName "kube-api-access-snhsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:17.554097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.554063    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:17.554097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:17.554093    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhsv\" (UniqueName: \"kubernetes.io/projected/a31b95ae-2c3e-4a99-bdd1-8b13546cca10-kube-api-access-snhsv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:18.044159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.044129    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"
Oct 30 03:30:18.044656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.044124    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml" event={"ID":"68216927-1902-41ec-a9ed-44264d7e064d","Type":"ContainerDied","Data":"5e743f4c142b12f6145b8f12f28ebec7558cf0431a1906fb58488a95228120df"}
Oct 30 03:30:18.044656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.044270    4877 scope.go:122] "RemoveContainer" containerID="47350dbdb082cb0fbe5e26460cbf8300d9fb54d777f5626c20e108869695d3c8"
Oct 30 03:30:18.045995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.045969    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"
Oct 30 03:30:18.046094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.046014    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx" event={"ID":"a31b95ae-2c3e-4a99-bdd1-8b13546cca10","Type":"ContainerDied","Data":"fad3eeb39833e6e31500f4d7cb1ae798760544c0c809eb22848feeb97eb6ec9e"}
Oct 30 03:30:18.055072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.055053    4877 scope.go:122] "RemoveContainer" containerID="e5a587d086b0b1213958c241af3357fc84264b0317112ca331e5da9849837710"
Oct 30 03:30:18.125776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.125737    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"]
Oct 30 03:30:18.136876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.136833    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888850-87kml"]
Oct 30 03:30:18.158257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.158223    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"]
Oct 30 03:30:18.162000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.161974    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888850-lflcx"]
Oct 30 03:30:18.170643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.170623    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"]
Oct 30 03:30:18.424503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.423723    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:18.465773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.465636    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca\") pod \"a969a7a1-7988-4819-9a9f-9caecf42b592\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") "
Oct 30 03:30:18.465773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.465709    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz\" (UniqueName: \"kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz\") pod \"a969a7a1-7988-4819-9a9f-9caecf42b592\" (UID: \"a969a7a1-7988-4819-9a9f-9caecf42b592\") "
Oct 30 03:30:18.520969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.520924    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca" pod "a969a7a1-7988-4819-9a9f-9caecf42b592" (UID: "a969a7a1-7988-4819-9a9f-9caecf42b592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:18.551078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.551022    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz" pod "a969a7a1-7988-4819-9a9f-9caecf42b592" (UID: "a969a7a1-7988-4819-9a9f-9caecf42b592"). InnerVolumeSpecName "kube-api-access-tg7hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:18.568379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.568338    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a969a7a1-7988-4819-9a9f-9caecf42b592-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:18.568379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.568377    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7hz\" (UniqueName: \"kubernetes.io/projected/a969a7a1-7988-4819-9a9f-9caecf42b592-kube-api-access-tg7hz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:18.704269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.704232    4877 scope.go:122] "RemoveContainer" containerID="f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36"
Oct 30 03:30:18.705466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:18.705444    4877 scope.go:122] "RemoveContainer" containerID="f1728342959df76d26a85180c68f0a7808e43d35378a893c184661946e618ed4"
Oct 30 03:30:19.057705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.057669    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt" event={"ID":"a969a7a1-7988-4819-9a9f-9caecf42b592","Type":"ContainerDied","Data":"7402b52393f4fc88dfc6b9078be9081a7f39794fbaf0646595893a6b19d9c309"}
Oct 30 03:30:19.058092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.057727    4877 scope.go:122] "RemoveContainer" containerID="772e13b50c4f4b8ba5fd625a62df8b60cf1d9cc31b5aa53ebbc49ff7d279cbc3"
Oct 30 03:30:19.058092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.057826    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"
Oct 30 03:30:19.121649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.121584    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"]
Oct 30 03:30:19.129264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.129237    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888850-bscdt"]
Oct 30 03:30:19.716825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.716777    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68216927-1902-41ec-a9ed-44264d7e064d" path="/var/lib/kubelet/pods/68216927-1902-41ec-a9ed-44264d7e064d/volumes"
Oct 30 03:30:19.717463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.717441    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" path="/var/lib/kubelet/pods/a31b95ae-2c3e-4a99-bdd1-8b13546cca10/volumes"
Oct 30 03:30:19.717978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:19.717959    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" path="/var/lib/kubelet/pods/a969a7a1-7988-4819-9a9f-9caecf42b592/volumes"
Oct 30 03:30:20.114457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.109682    4877 generic.go:356] "Generic (PLEG): container finished" podID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerID="644ac0baa187292716cf89aca9affb761c661e5351511538044b5060fd58c71b" exitCode=60
Oct 30 03:30:20.114457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.109841    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" event={"ID":"fbe83d78-b370-41f6-8ae0-d793d3fd3109","Type":"ContainerDied","Data":"644ac0baa187292716cf89aca9affb761c661e5351511538044b5060fd58c71b"}
Oct 30 03:30:20.114457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.109898    4877 scope.go:122] "RemoveContainer" containerID="f1728342959df76d26a85180c68f0a7808e43d35378a893c184661946e618ed4"
Oct 30 03:30:20.114457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.110352    4877 scope.go:122] "RemoveContainer" containerID="644ac0baa187292716cf89aca9affb761c661e5351511538044b5060fd58c71b"
Oct 30 03:30:20.114457 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:20.110706    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68_odh-ai-gateway-infra(fbe83d78-b370-41f6-8ae0-d793d3fd3109)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109"
Oct 30 03:30:20.121371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.121342    4877 generic.go:356] "Generic (PLEG): container finished" podID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerID="63cc56bc9bd974f825860140c3aa484db1836c76c92153d5f7eaf143d83fdfc4" exitCode=60
Oct 30 03:30:20.121517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.121450    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerDied","Data":"63cc56bc9bd974f825860140c3aa484db1836c76c92153d5f7eaf143d83fdfc4"}
Oct 30 03:30:20.121846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.121826    4877 scope.go:122] "RemoveContainer" containerID="63cc56bc9bd974f825860140c3aa484db1836c76c92153d5f7eaf143d83fdfc4"
Oct 30 03:30:20.122112 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:20.122087    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl_odh-ai-gateway-infra(7a7dc6f0-e183-4c06-ba47-f0c8073a15ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce"
Oct 30 03:30:20.154835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:20.145250    4877 scope.go:122] "RemoveContainer" containerID="f27c46a7f6ee13b49dc557c724312e4a3792620f206215299e5964126e398a36"
Oct 30 03:30:21.158293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.158253    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"]
Oct 30 03:30:21.292083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.292045    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"]
Oct 30 03:30:21.300631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.300606    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"]
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344052    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344084    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344110    4877 cpu_manager.go:414] "containerMap: removing container" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344118    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344135    4877 cpu_manager.go:414] "containerMap: removing container" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344145    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344165    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344172    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344183    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344190    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344202    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344209    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344241    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344249    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344441    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344459    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344469    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344479    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344488    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344497    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a969a7a1-7988-4819-9a9f-9caecf42b592" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344510    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344519    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:30:21.346568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.344529    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:30:21.350227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.350198    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.360750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.360673    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"]
Oct 30 03:30:21.369995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.369604    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:30:21.447462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.444645    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4z9\" (UniqueName: \"kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.447462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.444729    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.447462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.444755    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.534637 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.534603    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:21.545908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.545878    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4z9\" (UniqueName: \"kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.546511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.545963    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.546511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.545994    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.547131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.547111    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.548920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.548888    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.565093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.565070    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4z9\" (UniqueName: \"kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9\") pod \"maas-api-revoke-sub-75d5b356-tlh64\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.589831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.589801    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:21.647191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.647156    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca\") pod \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") "
Oct 30 03:30:21.647382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.647358    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c\" (UniqueName: \"kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c\") pod \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\" (UID: \"fbe83d78-b370-41f6-8ae0-d793d3fd3109\") "
Oct 30 03:30:21.648320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.648289    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca" pod "fbe83d78-b370-41f6-8ae0-d793d3fd3109" (UID: "fbe83d78-b370-41f6-8ae0-d793d3fd3109"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:21.654459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.649946    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c" pod "fbe83d78-b370-41f6-8ae0-d793d3fd3109" (UID: "fbe83d78-b370-41f6-8ae0-d793d3fd3109"). InnerVolumeSpecName "kube-api-access-hlk4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:21.692606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.692521    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:21.749073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.749030    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw\" (UniqueName: \"kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw\") pod \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") "
Oct 30 03:30:21.749287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.749146    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca\") pod \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\" (UID: \"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce\") "
Oct 30 03:30:21.749726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.749675    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe83d78-b370-41f6-8ae0-d793d3fd3109-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.749726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.749701    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlk4c\" (UniqueName: \"kubernetes.io/projected/fbe83d78-b370-41f6-8ae0-d793d3fd3109-kube-api-access-hlk4c\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.749889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.749857    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca" pod "7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" (UID: "7a7dc6f0-e183-4c06-ba47-f0c8073a15ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:21.753094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.753067    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw" pod "7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" (UID: "7a7dc6f0-e183-4c06-ba47-f0c8073a15ce"). InnerVolumeSpecName "kube-api-access-7mmmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:21.850628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.850591    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.850628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:21.850625    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmmw\" (UniqueName: \"kubernetes.io/projected/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce-kube-api-access-7mmmw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.002359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.002333    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"]
Oct 30 03:30:22.002613 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:30:22.002582    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4b45fc_5d63_4b71_900f_ebb76e4800ee.slice/crio-07a82b9713203ccaaec46a934ad176d5b1ba526dc90324f59133754d80e2f592 WatchSource:0}: Error finding container 07a82b9713203ccaaec46a934ad176d5b1ba526dc90324f59133754d80e2f592: Status 404 returned error can't find the container with id 07a82b9713203ccaaec46a934ad176d5b1ba526dc90324f59133754d80e2f592
Oct 30 03:30:22.159291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.159258    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"
Oct 30 03:30:22.159857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.159273    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68" event={"ID":"fbe83d78-b370-41f6-8ae0-d793d3fd3109","Type":"ContainerDied","Data":"733b41d8e64c28177c002c6180752da01f291a5fa157335b746088bc3db94dba"}
Oct 30 03:30:22.159990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.159900    4877 scope.go:122] "RemoveContainer" containerID="644ac0baa187292716cf89aca9affb761c661e5351511538044b5060fd58c71b"
Oct 30 03:30:22.163826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.163802    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"
Oct 30 03:30:22.163943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.163826    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl" event={"ID":"7a7dc6f0-e183-4c06-ba47-f0c8073a15ce","Type":"ContainerDied","Data":"079fcca9220e10bd688a1e0d4ae91cad5f2d356b31dd7fdb1c6c66ad371fef83"}
Oct 30 03:30:22.165624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.165586    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64" event={"ID":"8b4b45fc-5d63-4b71-900f-ebb76e4800ee","Type":"ContainerStarted","Data":"07a82b9713203ccaaec46a934ad176d5b1ba526dc90324f59133754d80e2f592"}
Oct 30 03:30:22.171581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.171378    4877 scope.go:122] "RemoveContainer" containerID="63cc56bc9bd974f825860140c3aa484db1836c76c92153d5f7eaf143d83fdfc4"
Oct 30 03:30:22.216651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.216622    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"]
Oct 30 03:30:22.222964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.222937    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888850-twrfl"]
Oct 30 03:30:22.262123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.262024    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"]
Oct 30 03:30:22.268162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:22.268128    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-dc3196-29888850-2jv68"]
Oct 30 03:30:23.179880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:23.179792    4877 generic.go:356] "Generic (PLEG): container finished" podID="8b4b45fc-5d63-4b71-900f-ebb76e4800ee" containerID="762f3cdc5c6f4e45192e7e5e322d7101d302bac131038c55391647301b6caafa" exitCode=0
Oct 30 03:30:23.180233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:23.179884    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64" event={"ID":"8b4b45fc-5d63-4b71-900f-ebb76e4800ee","Type":"ContainerDied","Data":"762f3cdc5c6f4e45192e7e5e322d7101d302bac131038c55391647301b6caafa"}
Oct 30 03:30:23.701805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:23.701767    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" path="/var/lib/kubelet/pods/7a7dc6f0-e183-4c06-ba47-f0c8073a15ce/volumes"
Oct 30 03:30:23.702213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:23.702198    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" path="/var/lib/kubelet/pods/fbe83d78-b370-41f6-8ae0-d793d3fd3109/volumes"
Oct 30 03:30:24.371810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.371775    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:24.493022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.492920    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca\") pod \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") "
Oct 30 03:30:24.493222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.493201    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca\") pod \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") "
Oct 30 03:30:24.493291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.493252    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9\" (UniqueName: \"kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9\") pod \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\" (UID: \"8b4b45fc-5d63-4b71-900f-ebb76e4800ee\") "
Oct 30 03:30:24.493418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.493386    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca" pod "8b4b45fc-5d63-4b71-900f-ebb76e4800ee" (UID: "8b4b45fc-5d63-4b71-900f-ebb76e4800ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:24.493871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.493842    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:24.495359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.495327    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca" pod "8b4b45fc-5d63-4b71-900f-ebb76e4800ee" (UID: "8b4b45fc-5d63-4b71-900f-ebb76e4800ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:24.495529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.495511    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9" pod "8b4b45fc-5d63-4b71-900f-ebb76e4800ee" (UID: "8b4b45fc-5d63-4b71-900f-ebb76e4800ee"). InnerVolumeSpecName "kube-api-access-km4z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:24.595718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.595673    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:24.595718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:24.595717    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4z9\" (UniqueName: \"kubernetes.io/projected/8b4b45fc-5d63-4b71-900f-ebb76e4800ee-kube-api-access-km4z9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:25.198348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.198253    4877 generic.go:356] "Generic (PLEG): container finished" podID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerID="e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72" exitCode=0
Oct 30 03:30:25.198348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.198334    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" event={"ID":"23a2a436-1bdd-48d4-b669-3eb5051cb190","Type":"ContainerDied","Data":"e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"}
Oct 30 03:30:25.198878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.198858    4877 scope.go:122] "RemoveContainer" containerID="e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"
Oct 30 03:30:25.199949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.199916    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64" event={"ID":"8b4b45fc-5d63-4b71-900f-ebb76e4800ee","Type":"ContainerDied","Data":"07a82b9713203ccaaec46a934ad176d5b1ba526dc90324f59133754d80e2f592"}
Oct 30 03:30:25.200025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.199960    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a82b9713203ccaaec46a934ad176d5b1ba526dc90324f59133754d80e2f592"
Oct 30 03:30:25.200025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.199917    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"
Oct 30 03:30:25.200488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.200458    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:30:25.200604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.200522    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:30:25.200604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:25.200557    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:30:26.211923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:26.211888    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" event={"ID":"23a2a436-1bdd-48d4-b669-3eb5051cb190","Type":"ContainerStarted","Data":"e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4"}
Oct 30 03:30:26.939228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:26.939194    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:30:26.939484 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:26.939463    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 03:30:26.939550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:26.939511    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 03:30:27.939253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:27.939215    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 03:30:27.939774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:27.939292    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 03:30:28.939231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:28.939191    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 03:30:28.939457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:28.939254    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 03:30:29.939322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:29.939285    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 03:30:29.939540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:29.939348    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 03:30:30.939790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:30.939751    4877 patch_prober.go:29] interesting pod/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 03:30:30.940199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:30.939814    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 03:30:31.944735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:31.944699    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:30:31.945246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:31.945224    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:30:31.946233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:31.946207    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:30:33.986884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:33.986789    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"]
Oct 30 03:30:38.987670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:38.987626    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v" podUID="3d5e6c09-a67a-476c-846a-12beb2307cac" containerName="payload-processing" containerID="cri-o://0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb" gracePeriod=25
Oct 30 03:30:39.269982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.269954    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:30:39.317357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.317327    4877 generic.go:356] "Generic (PLEG): container finished" podID="3d5e6c09-a67a-476c-846a-12beb2307cac" containerID="0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb" exitCode=0
Oct 30 03:30:39.317585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.317380    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"
Oct 30 03:30:39.317585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.317412    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v" event={"ID":"3d5e6c09-a67a-476c-846a-12beb2307cac","Type":"ContainerDied","Data":"0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb"}
Oct 30 03:30:39.317585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.317488    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v" event={"ID":"3d5e6c09-a67a-476c-846a-12beb2307cac","Type":"ContainerDied","Data":"6de76a98f8825e35703b6b9d655bc4c0df23d894ba0c79e670f5420ab52b1e75"}
Oct 30 03:30:39.317585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.317510    4877 scope.go:122] "RemoveContainer" containerID="0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb"
Oct 30 03:30:39.341390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.341368    4877 scope.go:122] "RemoveContainer" containerID="0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb"
Oct 30 03:30:39.341725 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:39.341707    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb\": container with ID starting with 0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb not found: ID does not exist" containerID="0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb"
Oct 30 03:30:39.341773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.341735    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb"} err="failed to get container status \"0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb\": rpc error: code = NotFound desc = could not find container \"0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb\": container with ID starting with 0d4cdfad138b087085b13a81aba899ca1f7b501387f36ad33182163e1fccd1eb not found: ID does not exist"
Oct 30 03:30:39.404080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.404049    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume\") pod \"3d5e6c09-a67a-476c-846a-12beb2307cac\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") "
Oct 30 03:30:39.404080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.404090    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk\" (UniqueName: \"kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk\") pod \"3d5e6c09-a67a-476c-846a-12beb2307cac\" (UID: \"3d5e6c09-a67a-476c-846a-12beb2307cac\") "
Oct 30 03:30:39.404545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.404515    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume" pod "3d5e6c09-a67a-476c-846a-12beb2307cac" (UID: "3d5e6c09-a67a-476c-846a-12beb2307cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:39.404749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.404727    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5e6c09-a67a-476c-846a-12beb2307cac-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:39.406324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.406300    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk" pod "3d5e6c09-a67a-476c-846a-12beb2307cac" (UID: "3d5e6c09-a67a-476c-846a-12beb2307cac"). InnerVolumeSpecName "kube-api-access-tdnqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:39.506587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.506500    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnqk\" (UniqueName: \"kubernetes.io/projected/3d5e6c09-a67a-476c-846a-12beb2307cac-kube-api-access-tdnqk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:39.664942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.664891    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"]
Oct 30 03:30:39.668724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.668697    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-65f21d-7c58f4bf4d-rr67v"]
Oct 30 03:30:39.710308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:39.710274    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5e6c09-a67a-476c-846a-12beb2307cac" path="/var/lib/kubelet/pods/3d5e6c09-a67a-476c-846a-12beb2307cac/volumes"
Oct 30 03:30:42.041506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:42.041472    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"]
Oct 30 03:30:42.041995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:42.041697    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" containerID="cri-o://5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0" gracePeriod=30
Oct 30 03:30:46.231197 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:46.231156    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:30:46.509531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:46.504717    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"]
Oct 30 03:30:46.509531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:46.504989    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy" containerID="cri-o://d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a" gracePeriod=30
Oct 30 03:30:47.116097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.116059    4877 patch_prober.go:29] interesting pod/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 03:30:47.116295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.116115    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 03:30:47.360087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.360057    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:30:47.406263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.406220    4877 generic.go:356] "Generic (PLEG): container finished" podID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerID="5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0" exitCode=0
Oct 30 03:30:47.406471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.406285    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"
Oct 30 03:30:47.406471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.406282    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" event={"ID":"0d9cf411-4e26-4c4c-9716-c7df36175bc2","Type":"ContainerDied","Data":"5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0"}
Oct 30 03:30:47.406471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.406395    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj" event={"ID":"0d9cf411-4e26-4c4c-9716-c7df36175bc2","Type":"ContainerDied","Data":"2bc870309cc82ef42429c156188ce6ed865ad5c56336feb7737f8c5b9cf92b63"}
Oct 30 03:30:47.406642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.406448    4877 scope.go:122] "RemoveContainer" containerID="5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0"
Oct 30 03:30:47.421670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.421646    4877 scope.go:122] "RemoveContainer" containerID="a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"
Oct 30 03:30:47.441833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.441805    4877 scope.go:122] "RemoveContainer" containerID="5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0"
Oct 30 03:30:47.442177 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:47.442154    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0\": container with ID starting with 5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0 not found: ID does not exist" containerID="5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0"
Oct 30 03:30:47.442267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.442189    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0"} err="failed to get container status \"5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0\": rpc error: code = NotFound desc = could not find container \"5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0\": container with ID starting with 5419195e3617b384241fa0b27fbb777a3ae62d8ad2f117a6cf492984804c2ab0 not found: ID does not exist"
Oct 30 03:30:47.442267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.442216    4877 scope.go:122] "RemoveContainer" containerID="a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"
Oct 30 03:30:47.442524 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:47.442492    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72\": container with ID starting with a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72 not found: ID does not exist" containerID="a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"
Oct 30 03:30:47.442607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.442541    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72"} err="failed to get container status \"a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72\": rpc error: code = NotFound desc = could not find container \"a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72\": container with ID starting with a654355fb1e57b7540519d774e22d0951eceb96d2f41147fe3427247fe6e3f72 not found: ID does not exist"
Oct 30 03:30:47.531379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531340    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531401    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531512    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531535    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531569    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531605    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531635    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531690    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.531895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531704    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:47.531895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531808    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket\") pod \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\" (UID: \"0d9cf411-4e26-4c4c-9716-c7df36175bc2\") "
Oct 30 03:30:47.532132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.531905    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:47.532222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532195    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:47.532280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532214    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:47.532344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532277    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:47.532438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532401    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.532508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532454    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.532508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532471    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.532508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532486    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.532508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.532502    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.533804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.533778    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:30:47.533905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.533823    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:47.533971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.533938    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "kube-api-access-wpnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:47.534028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.533963    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy" pod "0d9cf411-4e26-4c4c-9716-c7df36175bc2" (UID: "0d9cf411-4e26-4c4c-9716-c7df36175bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:47.633939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.633852    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpnkg\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-kube-api-access-wpnkg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.633939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.633884    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.633939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.633894    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.633939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.633902    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d9cf411-4e26-4c4c-9716-c7df36175bc2-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:47.748006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.747966    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"]
Oct 30 03:30:47.760103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:47.760067    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-194506-openshift-default-7fb974bc9c-zrkmj"]
Oct 30 03:30:49.699471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:49.699419    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" path="/var/lib/kubelet/pods/0d9cf411-4e26-4c4c-9716-c7df36175bc2/volumes"
Oct 30 03:30:51.808297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.808272    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:30:51.900587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900547    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900615    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900645    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900731    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900776    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900807    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900845    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900895    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900895    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900903    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900930    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:51.900964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.900957    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy\") pod \"fea18046-9d39-481e-bb13-bd5a848f0a23\" (UID: \"fea18046-9d39-481e-bb13-bd5a848f0a23\") "
Oct 30 03:30:51.901365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901245    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:51.901508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901470    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:51.901699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901679    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea18046-9d39-481e-bb13-bd5a848f0a23-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:51.901783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901704    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:51.901783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901717    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:51.901783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901730    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:51.901783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.901744    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:51.903179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.903153    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:51.903542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.903519    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "kube-api-access-t9j8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:51.903617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.903601    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:30:51.903780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:51.903761    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy" pod "fea18046-9d39-481e-bb13-bd5a848f0a23" (UID: "fea18046-9d39-481e-bb13-bd5a848f0a23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:52.002970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.002938    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.002970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.002966    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.003176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.002979    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9j8g\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-kube-api-access-t9j8g\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.003176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.002990    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea18046-9d39-481e-bb13-bd5a848f0a23-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:30:52.449311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.449278    4877 generic.go:356] "Generic (PLEG): container finished" podID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerID="d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a" exitCode=0
Oct 30 03:30:52.449531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.449340    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"
Oct 30 03:30:52.449531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.449372    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" event={"ID":"fea18046-9d39-481e-bb13-bd5a848f0a23","Type":"ContainerDied","Data":"d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a"}
Oct 30 03:30:52.449531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.449448    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4" event={"ID":"fea18046-9d39-481e-bb13-bd5a848f0a23","Type":"ContainerDied","Data":"293ee12ac837ea7c3aacebc2932bdb041ec55e6680f04329da40bff9423aaa72"}
Oct 30 03:30:52.449531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.449481    4877 scope.go:122] "RemoveContainer" containerID="d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a"
Oct 30 03:30:52.460430 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.460402    4877 scope.go:122] "RemoveContainer" containerID="5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"
Oct 30 03:30:52.470070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.470051    4877 scope.go:122] "RemoveContainer" containerID="d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a"
Oct 30 03:30:52.470355 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:52.470333    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a\": container with ID starting with d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a not found: ID does not exist" containerID="d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a"
Oct 30 03:30:52.470399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.470365    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a"} err="failed to get container status \"d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a\": rpc error: code = NotFound desc = could not find container \"d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a\": container with ID starting with d70a825e520d1deb3e8fbbdb973bcadc7bd732f8d6b7aa5153f9b1c57fd72b7a not found: ID does not exist"
Oct 30 03:30:52.470399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.470384    4877 scope.go:122] "RemoveContainer" containerID="5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"
Oct 30 03:30:52.470682 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:30:52.470662    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863\": container with ID starting with 5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863 not found: ID does not exist" containerID="5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"
Oct 30 03:30:52.470734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.470685    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863"} err="failed to get container status \"5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863\": rpc error: code = NotFound desc = could not find container \"5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863\": container with ID starting with 5a26b149ff80715bf6a6b536aac104084c4bda50ed06945d7ffabc33e5063863 not found: ID does not exist"
Oct 30 03:30:52.501811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.501780    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"]
Oct 30 03:30:52.513389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:52.513363    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-65f21d-openshift-default-869884d6b5-r6zk4"]
Oct 30 03:30:53.713162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:30:53.713124    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" path="/var/lib/kubelet/pods/fea18046-9d39-481e-bb13-bd5a848f0a23/volumes"
Oct 30 03:31:06.300927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:06.300892    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"]
Oct 30 03:31:06.301508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:06.301154    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy" containerID="cri-o://e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4" gracePeriod=30
Oct 30 03:31:11.579322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.579253    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:31:11.602107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.602072    4877 generic.go:356] "Generic (PLEG): container finished" podID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerID="e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4" exitCode=0
Oct 30 03:31:11.602253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.602129    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"
Oct 30 03:31:11.602253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.602137    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" event={"ID":"23a2a436-1bdd-48d4-b669-3eb5051cb190","Type":"ContainerDied","Data":"e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4"}
Oct 30 03:31:11.602253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.602229    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb" event={"ID":"23a2a436-1bdd-48d4-b669-3eb5051cb190","Type":"ContainerDied","Data":"ce7f97be264101e295b0589bd8e210f1b54f54fc962f1abbd56f10eba69196af"}
Oct 30 03:31:11.602371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.602260    4877 scope.go:122] "RemoveContainer" containerID="e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4"
Oct 30 03:31:11.617267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.617217    4877 scope.go:122] "RemoveContainer" containerID="e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"
Oct 30 03:31:11.629390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.629365    4877 scope.go:122] "RemoveContainer" containerID="e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4"
Oct 30 03:31:11.629685 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:11.629662    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4\": container with ID starting with e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4 not found: ID does not exist" containerID="e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4"
Oct 30 03:31:11.629817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.629695    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4"} err="failed to get container status \"e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4\": rpc error: code = NotFound desc = could not find container \"e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4\": container with ID starting with e46ac87fead606892b7e03cc731df69f371f85e10002e396a38be9a2068459d4 not found: ID does not exist"
Oct 30 03:31:11.629817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.629715    4877 scope.go:122] "RemoveContainer" containerID="e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"
Oct 30 03:31:11.629974 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:11.629957    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72\": container with ID starting with e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72 not found: ID does not exist" containerID="e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"
Oct 30 03:31:11.630015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.629978    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72"} err="failed to get container status \"e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72\": rpc error: code = NotFound desc = could not find container \"e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72\": container with ID starting with e1ce793bee7f31779d34a1458c236ec55ed3dc9170c33bf549501a7b2edaeb72 not found: ID does not exist"
Oct 30 03:31:11.684153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684118    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684196    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684270    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684308    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684348    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684440    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684437    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:11.684634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684476    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684568    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684643    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:11.684830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684696    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8\") pod \"23a2a436-1bdd-48d4-b669-3eb5051cb190\" (UID: \"23a2a436-1bdd-48d4-b669-3eb5051cb190\") "
Oct 30 03:31:11.684830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684772    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:11.684830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684800    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:11.685025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.684950    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:11.685284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.685183    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a2a436-1bdd-48d4-b669-3eb5051cb190-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.685284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.685208    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.685284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.685225    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.685284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.685241    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.685284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.685256    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.686776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.686750    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:11.687264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.687242    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:11.687522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.687504    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:11.687597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.687531    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8" pod "23a2a436-1bdd-48d4-b669-3eb5051cb190" (UID: "23a2a436-1bdd-48d4-b669-3eb5051cb190"). InnerVolumeSpecName "kube-api-access-4xgh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:11.786970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.786930    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.786970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.786963    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.786970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.786976    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xgh8\" (UniqueName: \"kubernetes.io/projected/23a2a436-1bdd-48d4-b669-3eb5051cb190-kube-api-access-4xgh8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.787268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.786989    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a2a436-1bdd-48d4-b669-3eb5051cb190-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:11.942529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.942433    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"]
Oct 30 03:31:11.946050 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:11.946021    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-773fbb-openshift-default-f69b4fdb5-6m9wb"]
Oct 30 03:31:13.698868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:13.698832    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" path="/var/lib/kubelet/pods/23a2a436-1bdd-48d4-b669-3eb5051cb190/volumes"
Oct 30 03:31:16.228924 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:16.228887    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:31:29.922892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:29.922849    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"]
Oct 30 03:31:29.923402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:29.923061    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn" podUID="6556ea60-dcf4-4020-bdb1-15fb07e71c82" containerName="discovery" containerID="cri-o://b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5" gracePeriod=30
Oct 30 03:31:30.215232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.215204    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:31:30.304785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.304742    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.304973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.304869    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.304973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.304920    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.305075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.305005    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.305075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.305068    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.305196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.305119    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.305258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.305197    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98\") pod \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\" (UID: \"6556ea60-dcf4-4020-bdb1-15fb07e71c82\") "
Oct 30 03:31:30.305627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.305589    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:30.307711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.307681    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.307711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.307699    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.307921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.307898    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:30.308074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.308048    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "kube-api-access-28b98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:30.308074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.308054    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:30.308501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.308483    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert" pod "6556ea60-dcf4-4020-bdb1-15fb07e71c82" (UID: "6556ea60-dcf4-4020-bdb1-15fb07e71c82"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:30.406512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406474    4877 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-dns-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.406512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406512    4877 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-csr-ca-configmap\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.406736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406528    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28b98\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-kube-api-access-28b98\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.406736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406543    4877 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-cacerts\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.406736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406559    4877 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6556ea60-dcf4-4020-bdb1-15fb07e71c82-local-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.406736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406574    4877 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-kubeconfig\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.406736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.406586    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6556ea60-dcf4-4020-bdb1-15fb07e71c82-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:30.741794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.741752    4877 generic.go:356] "Generic (PLEG): container finished" podID="6556ea60-dcf4-4020-bdb1-15fb07e71c82" containerID="b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5" exitCode=0
Oct 30 03:31:30.741980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.741812    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"
Oct 30 03:31:30.741980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.741835    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn" event={"ID":"6556ea60-dcf4-4020-bdb1-15fb07e71c82","Type":"ContainerDied","Data":"b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5"}
Oct 30 03:31:30.741980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.741897    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn" event={"ID":"6556ea60-dcf4-4020-bdb1-15fb07e71c82","Type":"ContainerDied","Data":"f7f5baec2244f7670e9d72add1ff6fb2220b85b6566c2537062f6b27423944e2"}
Oct 30 03:31:30.741980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.741922    4877 scope.go:122] "RemoveContainer" containerID="b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5"
Oct 30 03:31:30.753078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.753056    4877 scope.go:122] "RemoveContainer" containerID="b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5"
Oct 30 03:31:30.753468 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:30.753410    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5\": container with ID starting with b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5 not found: ID does not exist" containerID="b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5"
Oct 30 03:31:30.753584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.753478    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5"} err="failed to get container status \"b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5\": rpc error: code = NotFound desc = could not find container \"b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5\": container with ID starting with b3a417723c48d7d3f6c1fe2e0da36700ff9ed7bfc9e6febecf7565aa754de6d5 not found: ID does not exist"
Oct 30 03:31:30.809613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.809577    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"]
Oct 30 03:31:30.822498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:30.822457    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9p5qn"]
Oct 30 03:31:31.126556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.126518    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"]
Oct 30 03:31:31.127327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127309    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.127373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127329    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.127373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127354    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6556ea60-dcf4-4020-bdb1-15fb07e71c82" containerName="discovery"
Oct 30 03:31:31.127373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127363    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6556ea60-dcf4-4020-bdb1-15fb07e71c82" containerName="discovery"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127380    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127388    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127399    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127406    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127449    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127458    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127471    4877 cpu_manager.go:414] "containerMap: removing container" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127478    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127490    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.127505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127497    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127524    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127532    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127544    4877 cpu_manager.go:414] "containerMap: removing container" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127551    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68216927-1902-41ec-a9ed-44264d7e064d" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127567    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127574    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127585    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127592    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127610    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3d5e6c09-a67a-476c-846a-12beb2307cac" containerName="payload-processing"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127617    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5e6c09-a67a-476c-846a-12beb2307cac" containerName="payload-processing"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127658    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127666    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31b95ae-2c3e-4a99-bdd1-8b13546cca10" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127676    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8b4b45fc-5d63-4b71-900f-ebb76e4800ee" containerName="revoke-keys"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127683    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4b45fc-5d63-4b71-900f-ebb76e4800ee" containerName="revoke-keys"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127698    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127706    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127718    4877 cpu_manager.go:414] "containerMap: removing container" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy"
Oct 30 03:31:31.127811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127726    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127856    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127868    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127877    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127885    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5e6c09-a67a-476c-846a-12beb2307cac" containerName="payload-processing"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127898    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4b45fc-5d63-4b71-900f-ebb76e4800ee" containerName="revoke-keys"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127905    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127913    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127922    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127930    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6556ea60-dcf4-4020-bdb1-15fb07e71c82" containerName="discovery"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127939    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127947    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9cf411-4e26-4c4c-9716-c7df36175bc2" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127957    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a2a436-1bdd-48d4-b669-3eb5051cb190" containerName="istio-proxy"
Oct 30 03:31:31.128345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.127965    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:31:31.133269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.133247    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.149742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.149716    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3c874f-openshift-default-dockercfg-zt77d\""
Oct 30 03:31:31.152247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.152221    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"]
Oct 30 03:31:31.215471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215436    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215480    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215506    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215524    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215618    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215653    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215749    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215813    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.215948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.215866    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjjm\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.316909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.316866    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.316922    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.316963    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.316992    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317054    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317109    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317275    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317357    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317440    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjjm\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317476    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317802    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317867    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.317976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317878    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.318060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.317999    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.319588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.319568    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.319753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.319735    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.338634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.338602    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.338783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.338679    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjjm\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.447024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.446927    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:31.699046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.698965    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6556ea60-dcf4-4020-bdb1-15fb07e71c82" path="/var/lib/kubelet/pods/6556ea60-dcf4-4020-bdb1-15fb07e71c82/volumes"
Oct 30 03:31:31.771741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.771712    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"]
Oct 30 03:31:31.772019 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:31:31.771996    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28cb7f0_a115_442a_bdf3_d4969c0219b1.slice/crio-6633b76735789baac623370db69153b1025e8756621fb1ae9ddfeba2a19348d8 WatchSource:0}: Error finding container 6633b76735789baac623370db69153b1025e8756621fb1ae9ddfeba2a19348d8: Status 404 returned error can't find the container with id 6633b76735789baac623370db69153b1025e8756621fb1ae9ddfeba2a19348d8
Oct 30 03:31:31.774193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.774169    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:31:31.774258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.774226    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:31:31.774258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:31.774255    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:31:32.759125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:32.759082    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" event={"ID":"f28cb7f0-a115-442a-bdf3-d4969c0219b1","Type":"ContainerStarted","Data":"9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d"}
Oct 30 03:31:32.759599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:32.759151    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" event={"ID":"f28cb7f0-a115-442a-bdf3-d4969c0219b1","Type":"ContainerStarted","Data":"6633b76735789baac623370db69153b1025e8756621fb1ae9ddfeba2a19348d8"}
Oct 30 03:31:33.448163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:33.448126    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:33.450249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:33.450218    4877 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:31:33.450376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:33.450266    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 03:31:34.447519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:34.447482    4877 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:31:34.447937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:34.447542    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 03:31:34.509249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:34.509180    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podStartSLOduration=3.509159094 podStartE2EDuration="3.509159094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:32.82693236 +0000 UTC m=+1721.255821815" watchObservedRunningTime="2026-10-30 03:31:34.509159094 +0000 UTC m=+1722.938048514"
Oct 30 03:31:34.509510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:34.509471    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"]
Oct 30 03:31:34.509922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:34.509696    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" containerID="cri-o://226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a" gracePeriod=30
Oct 30 03:31:35.447537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:35.447499    4877 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:31:35.448026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:35.447558    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 03:31:36.448004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:36.447972    4877 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:31:36.448404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:36.448026    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 03:31:37.448263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:37.448228    4877 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 03:31:37.448663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:37.448276    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 03:31:37.606550 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:37.606506    4877 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:31:37.606743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:37.606577    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:31:39.490001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.489558    4877 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:31:39.490001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.489640    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:31:39.819233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.819197    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:39.820354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.820291    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:39.821100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.821079    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:31:39.824328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.824305    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:31:39.852545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.852514    4877 generic.go:356] "Generic (PLEG): container finished" podID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerID="226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a" exitCode=0
Oct 30 03:31:39.852679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.852564    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"
Oct 30 03:31:39.852679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.852595    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" event={"ID":"2d09e0b5-a925-4499-973c-85092ecdf66b","Type":"ContainerDied","Data":"226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a"}
Oct 30 03:31:39.852679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.852648    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg" event={"ID":"2d09e0b5-a925-4499-973c-85092ecdf66b","Type":"ContainerDied","Data":"b181e3805993215c6cff5af28717c4e1c0224527adad89100db33ac84921661d"}
Oct 30 03:31:39.852808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.852679    4877 scope.go:122] "RemoveContainer" containerID="226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a"
Oct 30 03:31:39.863057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.863038    4877 scope.go:122] "RemoveContainer" containerID="fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"
Oct 30 03:31:39.866709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.866688    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.866810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.866738    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.866810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.866803    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.866973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.866953    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:39.867140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.867105    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:39.867336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.867314    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:39.867547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.867528    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:39.868919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.868899    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "kube-api-access-ztlkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:39.873719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.873699    4877 scope.go:122] "RemoveContainer" containerID="226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a"
Oct 30 03:31:39.874009 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:39.873985    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a\": container with ID starting with 226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a not found: ID does not exist" containerID="226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a"
Oct 30 03:31:39.874110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.874020    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a"} err="failed to get container status \"226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a\": rpc error: code = NotFound desc = could not find container \"226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a\": container with ID starting with 226b50bb208a005fd55712eaa39d5b4ba49ea66f36688c3404c65f91f0e4403a not found: ID does not exist"
Oct 30 03:31:39.874110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.874046    4877 scope.go:122] "RemoveContainer" containerID="fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"
Oct 30 03:31:39.874297 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:39.874279    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2\": container with ID starting with fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2 not found: ID does not exist" containerID="fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"
Oct 30 03:31:39.874353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.874305    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2"} err="failed to get container status \"fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2\": rpc error: code = NotFound desc = could not find container \"fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2\": container with ID starting with fb1dd9c8553a5839e82ccf19f4ba88e704d22ebe8ef3404b0c2b9e77c9b93ca2 not found: ID does not exist"
Oct 30 03:31:39.967939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.967907    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.967939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.967951    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.968201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.967994    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.968201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.968042    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.968201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.968110    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.968201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.968184    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs\") pod \"2d09e0b5-a925-4499-973c-85092ecdf66b\" (UID: \"2d09e0b5-a925-4499-973c-85092ecdf66b\") "
Oct 30 03:31:39.968440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.968407    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:39.968665 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.968633    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:39.968754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.968671    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:39.969027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.969000    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d09e0b5-a925-4499-973c-85092ecdf66b-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:39.969127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.969034    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:39.969127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.969048    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:39.969127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.969062    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztlkb\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-kube-api-access-ztlkb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:39.970820 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.970793    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:39.971025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.970992    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:39.971804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:39.971780    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo" pod "2d09e0b5-a925-4499-973c-85092ecdf66b" (UID: "2d09e0b5-a925-4499-973c-85092ecdf66b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:40.069944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:40.069867    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:40.069944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:40.069899    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:40.069944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:40.069908    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d09e0b5-a925-4499-973c-85092ecdf66b-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:31:40.203435 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:40.203360    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"]
Oct 30 03:31:40.211475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:40.209727    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-6w5wg"]
Oct 30 03:31:41.697989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:31:41.697955    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" path="/var/lib/kubelet/pods/2d09e0b5-a925-4499-973c-85092ecdf66b/volumes"
Oct 30 03:31:46.229487 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:31:46.229452    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:32:16.228938 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:32:16.228901    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:32:40.042279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:40.042241    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"]
Oct 30 03:32:40.053715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:40.053686    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8314ed-nrprr"]
Oct 30 03:32:41.697880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:41.697846    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cbaf32-7556-4afd-96e4-dfd19b95d866" path="/var/lib/kubelet/pods/f3cbaf32-7556-4afd-96e4-dfd19b95d866/volumes"
Oct 30 03:32:46.228221 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:32:46.228181    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:32:52.173122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:52.173090    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:32:52.204979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:52.204955    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:32:54.695792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:54.695765    4877 scope.go:122] "RemoveContainer" containerID="80d8da0e70675c273c7f07ef6afcfa830c80dceca4aae24c0799bb0aac14f78c"
Oct 30 03:32:57.029356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:57.029319    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"]
Oct 30 03:32:57.034266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:57.034241    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b158d8b6-jfc72"]
Oct 30 03:32:57.699064 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:32:57.699030    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e615aebb-ce42-48a3-9885-6cd9c6ec6059" path="/var/lib/kubelet/pods/e615aebb-ce42-48a3-9885-6cd9c6ec6059/volumes"
Oct 30 03:33:02.029385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:02.029348    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"]
Oct 30 03:33:02.041045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:02.041016    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7393321c-qkdlr"]
Oct 30 03:33:03.698290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:03.698256    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f588f24-b25b-47ca-ac28-f1e2231d7c10" path="/var/lib/kubelet/pods/6f588f24-b25b-47ca-ac28-f1e2231d7c10/volumes"
Oct 30 03:33:10.043618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:10.043585    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"]
Oct 30 03:33:10.055344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:10.055314    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bbea6be-dsvhh"]
Oct 30 03:33:11.698469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:11.698436    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6898126-b2a3-49cb-b21d-03073fbc7ea8" path="/var/lib/kubelet/pods/f6898126-b2a3-49cb-b21d-03073fbc7ea8/volumes"
Oct 30 03:33:15.028651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:15.028621    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"]
Oct 30 03:33:15.040238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:15.040214    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c29f60-xlfmb"]
Oct 30 03:33:15.697951 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:15.697910    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76a156c-822f-4067-9b42-1aa4643e1923" path="/var/lib/kubelet/pods/c76a156c-822f-4067-9b42-1aa4643e1923/volumes"
Oct 30 03:33:16.227779 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:33:16.227742    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:33:23.029493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:23.029460    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"]
Oct 30 03:33:23.040909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:23.040884    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee60a867-zr8px"]
Oct 30 03:33:23.699066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:23.699026    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0f9141-2238-4a84-bf8b-45c1d15f2ad8" path="/var/lib/kubelet/pods/db0f9141-2238-4a84-bf8b-45c1d15f2ad8/volumes"
Oct 30 03:33:34.033359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:34.033137    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"]
Oct 30 03:33:34.043623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:34.043594    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad9f014-g47xh"]
Oct 30 03:33:35.697741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:35.697702    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e55ebd-8a02-4595-bcc1-d3625030f099" path="/var/lib/kubelet/pods/a9e55ebd-8a02-4595-bcc1-d3625030f099/volumes"
Oct 30 03:33:39.038414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:39.038377    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"]
Oct 30 03:33:39.050658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:39.050624    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac57b6c6-pbwhl"]
Oct 30 03:33:39.705293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:39.705256    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695e9334-afb6-4bbc-81a2-21170fd676c5" path="/var/lib/kubelet/pods/695e9334-afb6-4bbc-81a2-21170fd676c5/volumes"
Oct 30 03:33:46.228929 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:33:46.228889    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:33:47.027577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:47.027542    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"]
Oct 30 03:33:47.039207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:47.039177    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd15d502-v86tk"]
Oct 30 03:33:47.698398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:47.698354    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef827b08-faae-48a4-969e-e15a9f5101c0" path="/var/lib/kubelet/pods/ef827b08-faae-48a4-969e-e15a9f5101c0/volumes"
Oct 30 03:33:54.734956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:54.734923    4877 scope.go:122] "RemoveContainer" containerID="b047ca7e92f040439c9afb39aa28b1ff5f1ce08aeb12c06656a560c1b952e193"
Oct 30 03:33:54.745287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:33:54.745265    4877 scope.go:122] "RemoveContainer" containerID="5c9996afb7434b58a6aff9cb4de3e4ac5a1e7880799a01cd14d77faede4bbc86"
Oct 30 03:34:16.229749 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:34:16.229686    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:34:18.616133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.616096    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"]
Oct 30 03:34:18.616962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.616942    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy"
Oct 30 03:34:18.617009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.616965    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea18046-9d39-481e-bb13-bd5a848f0a23" containerName="istio-proxy"
Oct 30 03:34:18.617009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.616994    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy"
Oct 30 03:34:18.617009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617002    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy"
Oct 30 03:34:18.617113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617013    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy"
Oct 30 03:34:18.617113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617021    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy"
Oct 30 03:34:18.617191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617179    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe83d78-b370-41f6-8ae0-d793d3fd3109" containerName="cleanup"
Oct 30 03:34:18.617228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617200    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7dc6f0-e183-4c06-ba47-f0c8073a15ce" containerName="cleanup"
Oct 30 03:34:18.617228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617210    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy"
Oct 30 03:34:18.617228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.617219    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d09e0b5-a925-4499-973c-85092ecdf66b" containerName="istio-proxy"
Oct 30 03:34:18.621147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.621125    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.636161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.636137    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"]
Oct 30 03:34:18.644279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.644258    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:34:18.644400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.644379    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:34:18.736303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.736257    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.736539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.736398    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvgx\" (UniqueName: \"kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.736598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.736559    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.837223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.837188    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.837476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.837369    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.837476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.837409    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvgx\" (UniqueName: \"kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.838070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.838049    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.839786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.839758    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.855841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.855819    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvgx\" (UniqueName: \"kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx\") pod \"maas-api-revoke-sub-5d09dbda-kc9lk\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:18.936067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:18.935991    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:19.250554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:19.250524    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"]
Oct 30 03:34:19.251182 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:34:19.251156    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf813ab01_ee7f_4519_9237_7b641466dc3c.slice/crio-6a8f7a71ca0385aeab088ce5315984ec63f05cd72f4e24fc2cc5c797bc06126a WatchSource:0}: Error finding container 6a8f7a71ca0385aeab088ce5315984ec63f05cd72f4e24fc2cc5c797bc06126a: Status 404 returned error can't find the container with id 6a8f7a71ca0385aeab088ce5315984ec63f05cd72f4e24fc2cc5c797bc06126a
Oct 30 03:34:19.982321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:19.982198    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk" event={"ID":"f813ab01-ee7f-4519-9237-7b641466dc3c","Type":"ContainerStarted","Data":"6f7d75549689c9bcb4b030ee392ba396f27c8e56a93f8944a769c27b2c0bfa84"}
Oct 30 03:34:19.982321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:19.982261    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk" event={"ID":"f813ab01-ee7f-4519-9237-7b641466dc3c","Type":"ContainerStarted","Data":"6a8f7a71ca0385aeab088ce5315984ec63f05cd72f4e24fc2cc5c797bc06126a"}
Oct 30 03:34:20.024581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:20.024519    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk" podStartSLOduration=2.024503828 podStartE2EDuration="2.024503828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:20.020826474 +0000 UTC m=+1888.449715892" watchObservedRunningTime="2026-10-30 03:34:20.024503828 +0000 UTC m=+1888.453393272"
Oct 30 03:34:20.992233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:20.992195    4877 generic.go:356] "Generic (PLEG): container finished" podID="f813ab01-ee7f-4519-9237-7b641466dc3c" containerID="6f7d75549689c9bcb4b030ee392ba396f27c8e56a93f8944a769c27b2c0bfa84" exitCode=0
Oct 30 03:34:20.992653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:20.992276    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk" event={"ID":"f813ab01-ee7f-4519-9237-7b641466dc3c","Type":"ContainerDied","Data":"6f7d75549689c9bcb4b030ee392ba396f27c8e56a93f8944a769c27b2c0bfa84"}
Oct 30 03:34:22.143723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.143699    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:22.280047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.280015    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca\") pod \"f813ab01-ee7f-4519-9237-7b641466dc3c\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") "
Oct 30 03:34:22.280237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.280198    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca\") pod \"f813ab01-ee7f-4519-9237-7b641466dc3c\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") "
Oct 30 03:34:22.280354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.280338    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx\" (UniqueName: \"kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx\") pod \"f813ab01-ee7f-4519-9237-7b641466dc3c\" (UID: \"f813ab01-ee7f-4519-9237-7b641466dc3c\") "
Oct 30 03:34:22.280460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.280336    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca" pod "f813ab01-ee7f-4519-9237-7b641466dc3c" (UID: "f813ab01-ee7f-4519-9237-7b641466dc3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:22.280787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.280767    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:22.282303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.282281    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca" pod "f813ab01-ee7f-4519-9237-7b641466dc3c" (UID: "f813ab01-ee7f-4519-9237-7b641466dc3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:22.282666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.282649    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx" pod "f813ab01-ee7f-4519-9237-7b641466dc3c" (UID: "f813ab01-ee7f-4519-9237-7b641466dc3c"). InnerVolumeSpecName "kube-api-access-prvgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:22.382382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.382349    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f813ab01-ee7f-4519-9237-7b641466dc3c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:22.382382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:22.382380    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvgx\" (UniqueName: \"kubernetes.io/projected/f813ab01-ee7f-4519-9237-7b641466dc3c-kube-api-access-prvgx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.007957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:23.007927    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"
Oct 30 03:34:23.008126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:23.007924    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk" event={"ID":"f813ab01-ee7f-4519-9237-7b641466dc3c","Type":"ContainerDied","Data":"6a8f7a71ca0385aeab088ce5315984ec63f05cd72f4e24fc2cc5c797bc06126a"}
Oct 30 03:34:23.008126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:23.008068    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8f7a71ca0385aeab088ce5315984ec63f05cd72f4e24fc2cc5c797bc06126a"
Oct 30 03:34:28.746012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:28.745977    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"]
Oct 30 03:34:28.746470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:28.746180    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy" containerID="cri-o://3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25" gracePeriod=30
Oct 30 03:34:28.758339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:28.758309    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"]
Oct 30 03:34:28.758563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:28.758541    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy" containerID="cri-o://c0bcd926818638e28f0ec60e586dd3b8645ad4b944b0ad4fdb44e759f6fb569c" gracePeriod=30
Oct 30 03:34:28.761374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:28.761348    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"]
Oct 30 03:34:28.761636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:28.761578    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy" containerID="cri-o://b5cbe3c6bf179a3270c5f136f8e4796eba4e10b7de9121b505f451a93b96bfc8" gracePeriod=30
Oct 30 03:34:32.126947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:32.126904    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"]
Oct 30 03:34:32.127401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:32.127165    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy" containerID="cri-o://0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60" gracePeriod=30
Oct 30 03:34:34.044098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.044067    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:34:34.093141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.093100    4877 generic.go:356] "Generic (PLEG): container finished" podID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerID="b5cbe3c6bf179a3270c5f136f8e4796eba4e10b7de9121b505f451a93b96bfc8" exitCode=0
Oct 30 03:34:34.093275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.093179    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" event={"ID":"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c","Type":"ContainerDied","Data":"b5cbe3c6bf179a3270c5f136f8e4796eba4e10b7de9121b505f451a93b96bfc8"}
Oct 30 03:34:34.095011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.094988    4877 generic.go:356] "Generic (PLEG): container finished" podID="9a26cda8-0568-4e08-af78-c239f8139068" containerID="c0bcd926818638e28f0ec60e586dd3b8645ad4b944b0ad4fdb44e759f6fb569c" exitCode=0
Oct 30 03:34:34.095124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.095063    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" event={"ID":"9a26cda8-0568-4e08-af78-c239f8139068","Type":"ContainerDied","Data":"c0bcd926818638e28f0ec60e586dd3b8645ad4b944b0ad4fdb44e759f6fb569c"}
Oct 30 03:34:34.095124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.095120    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf" event={"ID":"9a26cda8-0568-4e08-af78-c239f8139068","Type":"ContainerDied","Data":"7c76c4bf1e3c62a197eeb13d8a7aafd1a9494c7f9b7e3ce8bc9d9b7d088885f4"}
Oct 30 03:34:34.095240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.095143    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c76c4bf1e3c62a197eeb13d8a7aafd1a9494c7f9b7e3ce8bc9d9b7d088885f4"
Oct 30 03:34:34.097161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.097039    4877 generic.go:356] "Generic (PLEG): container finished" podID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerID="3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25" exitCode=0
Oct 30 03:34:34.105541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.098416    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"
Oct 30 03:34:34.105541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.103822    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:34:34.105790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.105766    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" event={"ID":"77492b62-38b7-40d7-a7ef-1cda9ae8d778","Type":"ContainerDied","Data":"3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25"}
Oct 30 03:34:34.108683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.108646    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95" event={"ID":"77492b62-38b7-40d7-a7ef-1cda9ae8d778","Type":"ContainerDied","Data":"97b15a9c6ae27cafa02a9ecf495caa30eb85834cc945ad1ef6d7d3b119c853bb"}
Oct 30 03:34:34.108904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.108827    4877 scope.go:122] "RemoveContainer" containerID="3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25"
Oct 30 03:34:34.126459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.122619    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:34:34.126459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.125781    4877 scope.go:122] "RemoveContainer" containerID="3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25"
Oct 30 03:34:34.126459 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:34:34.126114    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25\": container with ID starting with 3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25 not found: ID does not exist" containerID="3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25"
Oct 30 03:34:34.126459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.126139    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25"} err="failed to get container status \"3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25\": rpc error: code = NotFound desc = could not find container \"3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25\": container with ID starting with 3fc43f2938cd558ad748cc67fbb46d7f85a509829988bff5155cdd6377591d25 not found: ID does not exist"
Oct 30 03:34:34.223267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223235    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.223267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223268    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223296    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223315    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223337    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223373    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223394    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223453    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223477    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223511    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.223568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223566    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223592    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223634    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223682    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223722    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223762    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223808    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223836    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223908    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223960    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.223966    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.224070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224008    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224082    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224111    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo\") pod \"9a26cda8-0568-4e08-af78-c239f8139068\" (UID: \"9a26cda8-0568-4e08-af78-c239f8139068\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224163    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224175    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224249    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224286    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy\") pod \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\" (UID: \"77492b62-38b7-40d7-a7ef-1cda9ae8d778\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224321    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket\") pod \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\" (UID: \"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c\") "
Oct 30 03:34:34.224691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224476    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:34.225125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224991    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.225125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.224999    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.225241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.225184    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.225319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.225295    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:34.225500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.225380    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.225612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.225584    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.225681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.225650    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:34.226145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226123    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226280    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226393    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226504    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226596    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226677    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226762    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a26cda8-0568-4e08-af78-c239f8139068-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226838    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.226926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226915    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.227006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.226996    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.227198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.227169    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "kube-api-access-gpmx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.227864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.227537    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.227864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.227603    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.227864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.227794    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.228220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.227954    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.228220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.228111    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.228220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.228165    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:34:34.229650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.229617    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.229961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.229928    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.231317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.231280    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.231670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.231645    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "kube-api-access-rd5pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.231766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.231669    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "kube-api-access-hrzxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:34.231937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.231900    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:34:34.231937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.231908    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:34:34.231937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.231917    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy" pod "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" (UID: "0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.233572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.233548    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy" pod "77492b62-38b7-40d7-a7ef-1cda9ae8d778" (UID: "77492b62-38b7-40d7-a7ef-1cda9ae8d778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.233675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.233605    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy" pod "9a26cda8-0568-4e08-af78-c239f8139068" (UID: "9a26cda8-0568-4e08-af78-c239f8139068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:34.327988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.327952    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.327988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.327984    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.327988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.327994    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpmx4\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-kube-api-access-gpmx4\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328004    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328014    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328022    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a26cda8-0568-4e08-af78-c239f8139068-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328029    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328037    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a26cda8-0568-4e08-af78-c239f8139068-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328045    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328052    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd5pb\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-kube-api-access-rd5pb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328061    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328069    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a26cda8-0568-4e08-af78-c239f8139068-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328077    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328086    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrzxm\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-kube-api-access-hrzxm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328094    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77492b62-38b7-40d7-a7ef-1cda9ae8d778-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328102    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.328239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.328109    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:34.446255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.446224    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"]
Oct 30 03:34:34.449287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:34.449248    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8xm95"]
Oct 30 03:34:35.106596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.106519    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"
Oct 30 03:34:35.106596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.106541    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"
Oct 30 03:34:35.106596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.106535    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn" event={"ID":"0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c","Type":"ContainerDied","Data":"c54ddd04c47e2d5920518d4fcdd0cc63697a7f4e0487eee88808ed38cd69f2ca"}
Oct 30 03:34:35.106596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.106594    4877 scope.go:122] "RemoveContainer" containerID="b5cbe3c6bf179a3270c5f136f8e4796eba4e10b7de9121b505f451a93b96bfc8"
Oct 30 03:34:35.169380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.169343    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"]
Oct 30 03:34:35.172781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.172754    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-z5jpf"]
Oct 30 03:34:35.204525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.204487    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"]
Oct 30 03:34:35.207449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.207412    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-8pwsn"]
Oct 30 03:34:35.698524 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.698490    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" path="/var/lib/kubelet/pods/0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c/volumes"
Oct 30 03:34:35.698955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.698941    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" path="/var/lib/kubelet/pods/77492b62-38b7-40d7-a7ef-1cda9ae8d778/volumes"
Oct 30 03:34:35.699360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:35.699346    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a26cda8-0568-4e08-af78-c239f8139068" path="/var/lib/kubelet/pods/9a26cda8-0568-4e08-af78-c239f8139068/volumes"
Oct 30 03:34:37.413270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.413204    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:34:37.461706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.461659    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.461991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.461967    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462162    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462305    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462491    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462627    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462172    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:37.462659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462344    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:37.462914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462630    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:37.462914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462681    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462710    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.462914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.462838    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo\") pod \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\" (UID: \"ae3a22e1-6bcf-405d-b714-d4e6d131e148\") "
Oct 30 03:34:37.463140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.463051    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:37.463284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.463263    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.463343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.463289    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.464112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.464084    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:37.465374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.463304    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.465553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.465384    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.466215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.466108    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:34:37.466821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.466497    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:37.466821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.466626    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "kube-api-access-7qtht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:37.471956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.471891    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token" pod "ae3a22e1-6bcf-405d-b714-d4e6d131e148" (UID: "ae3a22e1-6bcf-405d-b714-d4e6d131e148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:37.489148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.489102    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"]
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490662    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f813ab01-ee7f-4519-9237-7b641466dc3c" containerName="revoke-keys"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490688    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f813ab01-ee7f-4519-9237-7b641466dc3c" containerName="revoke-keys"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490712    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490720    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490761    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490771    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490792    4877 cpu_manager.go:414] "containerMap: removing container" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490800    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490828    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490835    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490962    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0e9eaa-3db7-4cd1-91dd-0653ce9f4a8c" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490974    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="77492b62-38b7-40d7-a7ef-1cda9ae8d778" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.490991    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a26cda8-0568-4e08-af78-c239f8139068" containerName="istio-proxy"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.491002    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f813ab01-ee7f-4519-9237-7b641466dc3c" containerName="revoke-keys"
Oct 30 03:34:37.491743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.491012    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerName="istio-proxy"
Oct 30 03:34:37.496477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.495529    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.516520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.511761    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-da4553-openshift-default-dockercfg-2mt2c\""
Oct 30 03:34:37.520980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.520956    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"]
Oct 30 03:34:37.566751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.566717    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.566945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.566887    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.566945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.566922    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.566949    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tmb\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567049    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567108    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567177    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567241    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567258    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.567375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567329    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.567375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567340    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.567375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567349    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtht\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-kube-api-access-7qtht\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.567375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567358    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a22e1-6bcf-405d-b714-d4e6d131e148-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.567375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.567366    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a22e1-6bcf-405d-b714-d4e6d131e148-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.668635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668543    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668590    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668640    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668662    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668695    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668789    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668812    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.668883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668844    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tmb\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.669180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.668917    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.669180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.669004    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.669180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.669025    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.669335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.669270    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.669335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.669314    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.669739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.669719    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.670890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.670861    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.671496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.671469    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.687841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.687813    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.687958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.687925    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tmb\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.822223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.822178    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:37.889250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.889204    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"]
Oct 30 03:34:37.930146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.929711    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"]
Oct 30 03:34:37.930146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.929853    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.966774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.966742    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-1a1b45-openshift-default-dockercfg-gx88g\""
Oct 30 03:34:37.973146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973116    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973284    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973399    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2d5\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973505    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973540    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973612    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973645 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973634    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973692    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:37.973757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:37.973715    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074264    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074299    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074458    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074500    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074553    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074603    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074632    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2d5\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074778    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074827    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.074892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.074848    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.075189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.075165    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.075290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.075204    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.075290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.075217    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.076921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.076892    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.077235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.077217    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.093022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.092989    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.093356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.093329    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2d5\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.131074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.131033    4877 generic.go:356] "Generic (PLEG): container finished" podID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" containerID="0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60" exitCode=0
Oct 30 03:34:38.131218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.131073    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" event={"ID":"ae3a22e1-6bcf-405d-b714-d4e6d131e148","Type":"ContainerDied","Data":"0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60"}
Oct 30 03:34:38.131218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.131149    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h" event={"ID":"ae3a22e1-6bcf-405d-b714-d4e6d131e148","Type":"ContainerDied","Data":"cd1e0b6aee46b86bb102cead66d767c40a278330bef64443aac37c147f60ef6f"}
Oct 30 03:34:38.131218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.131176    4877 scope.go:122] "RemoveContainer" containerID="0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60"
Oct 30 03:34:38.131218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.131193    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"
Oct 30 03:34:38.141886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.141870    4877 scope.go:122] "RemoveContainer" containerID="0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60"
Oct 30 03:34:38.142174 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:34:38.142155    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60\": container with ID starting with 0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60 not found: ID does not exist" containerID="0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60"
Oct 30 03:34:38.142255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.142182    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60"} err="failed to get container status \"0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60\": rpc error: code = NotFound desc = could not find container \"0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60\": container with ID starting with 0b26d046399d684ff849f210677c965c4e5ab90805f718bc14ee55f74d9b3f60 not found: ID does not exist"
Oct 30 03:34:38.181640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.181556    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"]
Oct 30 03:34:38.186208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.186177    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-8cq5h"]
Oct 30 03:34:38.191791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.191768    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"]
Oct 30 03:34:38.192313 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:34:38.192287    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fa1ab1_f125_46d3_9836_9bc979fe9cd3.slice/crio-4c373c148f1004e39aea3a416408a6b8584a028ba816a1a4c77b60276ba47b77 WatchSource:0}: Error finding container 4c373c148f1004e39aea3a416408a6b8584a028ba816a1a4c77b60276ba47b77: Status 404 returned error can't find the container with id 4c373c148f1004e39aea3a416408a6b8584a028ba816a1a4c77b60276ba47b77
Oct 30 03:34:38.194855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.194823    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:38.194950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.194885    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:38.194950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.194912    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:38.261703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.261677    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:38.590215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.590178    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"]
Oct 30 03:34:38.593438 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:34:38.593393    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6f100b_2891_446b_ab4d_5b7f535e9e00.slice/crio-d5f5d41066c73e4452240c478797b5e65d465b141b93cf5d6def6954b5538445 WatchSource:0}: Error finding container d5f5d41066c73e4452240c478797b5e65d465b141b93cf5d6def6954b5538445: Status 404 returned error can't find the container with id d5f5d41066c73e4452240c478797b5e65d465b141b93cf5d6def6954b5538445
Oct 30 03:34:38.596028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.595994    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:38.596140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.596083    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:38.596140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:38.596127    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:39.139831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.139785    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" event={"ID":"ad6f100b-2891-446b-ab4d-5b7f535e9e00","Type":"ContainerStarted","Data":"d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"}
Oct 30 03:34:39.140061 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.139846    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" event={"ID":"ad6f100b-2891-446b-ab4d-5b7f535e9e00","Type":"ContainerStarted","Data":"d5f5d41066c73e4452240c478797b5e65d465b141b93cf5d6def6954b5538445"}
Oct 30 03:34:39.141635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.141602    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" event={"ID":"97fa1ab1-f125-46d3-9836-9bc979fe9cd3","Type":"ContainerStarted","Data":"c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"}
Oct 30 03:34:39.141766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.141662    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" event={"ID":"97fa1ab1-f125-46d3-9836-9bc979fe9cd3","Type":"ContainerStarted","Data":"4c373c148f1004e39aea3a416408a6b8584a028ba816a1a4c77b60276ba47b77"}
Oct 30 03:34:39.183653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.183602    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podStartSLOduration=2.183587994 podStartE2EDuration="2.183587994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:39.18008633 +0000 UTC m=+1907.608975748" watchObservedRunningTime="2026-10-30 03:34:39.183587994 +0000 UTC m=+1907.612477412"
Oct 30 03:34:39.223860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.223807    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podStartSLOduration=2.223793474 podStartE2EDuration="2.223793474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:39.220772385 +0000 UTC m=+1907.649661818" watchObservedRunningTime="2026-10-30 03:34:39.223793474 +0000 UTC m=+1907.652682891"
Oct 30 03:34:39.262395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.262363    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:39.268097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.268073    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:39.699147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.699112    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3a22e1-6bcf-405d-b714-d4e6d131e148" path="/var/lib/kubelet/pods/ae3a22e1-6bcf-405d-b714-d4e6d131e148/volumes"
Oct 30 03:34:39.822585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.822555    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:39.827561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:39.827535    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:40.151286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:40.151243    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:40.151286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:40.151298    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:40.151943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:40.151919    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:34:40.152054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:40.151995    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:34:44.917006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.916967    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"]
Oct 30 03:34:44.920875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.920854    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:44.977446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.955850    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"]
Oct 30 03:34:44.977446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.959179    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss245\" (UniqueName: \"kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245\") pod \"payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:44.977446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.959267    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:44.977446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.974877    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-1a1b45\""
Oct 30 03:34:44.977446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:44.975052    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-1a1b45-dockercfg-qhb6b\""
Oct 30 03:34:45.060019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:45.059980    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss245\" (UniqueName: \"kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245\") pod \"payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:45.060200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:45.060034    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:45.060743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:45.060720    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:45.078624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:45.078593    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss245\" (UniqueName: \"kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245\") pod \"payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:45.236357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:45.236262    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:45.556782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:45.556749    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"]
Oct 30 03:34:45.557490 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:34:45.557462    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baa73a5_9e5f_4aa3_aaea_97c02c952e3a.slice/crio-fe0cd9da5ad3789b094ae09d8eb04be39b47a05d1f15781c9261135ca2b53e5a WatchSource:0}: Error finding container fe0cd9da5ad3789b094ae09d8eb04be39b47a05d1f15781c9261135ca2b53e5a: Status 404 returned error can't find the container with id fe0cd9da5ad3789b094ae09d8eb04be39b47a05d1f15781c9261135ca2b53e5a
Oct 30 03:34:46.197512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:46.197472    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" event={"ID":"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a","Type":"ContainerStarted","Data":"e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62"}
Oct 30 03:34:46.197997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:46.197535    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" event={"ID":"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a","Type":"ContainerStarted","Data":"fe0cd9da5ad3789b094ae09d8eb04be39b47a05d1f15781c9261135ca2b53e5a"}
Oct 30 03:34:46.197997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:46.197587    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:46.232015 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:34:46.231977    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:34:46.237594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:46.237547    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" podStartSLOduration=2.23753237 podStartE2EDuration="2.23753237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:46.234623844 +0000 UTC m=+1914.663513265" watchObservedRunningTime="2026-10-30 03:34:46.23753237 +0000 UTC m=+1914.666421788"
Oct 30 03:34:53.534746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.534714    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"]
Oct 30 03:34:53.540273 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.540241    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.572390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.572359    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"]
Oct 30 03:34:53.578221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.577167    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7d99a4-openshift-default-dockercfg-wfkbf\""
Oct 30 03:34:53.649076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.649037    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.649475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.649412    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.649699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.649681    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.649851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.649836    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.650387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.650360    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.651119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.651097    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.651359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.651341    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.652271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.652229    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.653015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.652975    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpr8l\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755126    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755283    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpr8l\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755391    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755452    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755480    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755495    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755596    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755595    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755646    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755715    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755849    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.755938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.755892    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.756131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.756108    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.756263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.756249    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.757704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.757684    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.758023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.758007    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.773705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.773685    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.773955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.773936    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpr8l\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:53.860639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:53.860562    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:54.188037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.187999    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"]
Oct 30 03:34:54.189263 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:34:54.189227    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa525320_3c49_4e12_badd_234109092779.slice/crio-1194f52b3092704720774074b4d8d7ed7eb531177ebd20a52ef4645cab2dc38f WatchSource:0}: Error finding container 1194f52b3092704720774074b4d8d7ed7eb531177ebd20a52ef4645cab2dc38f: Status 404 returned error can't find the container with id 1194f52b3092704720774074b4d8d7ed7eb531177ebd20a52ef4645cab2dc38f
Oct 30 03:34:54.191784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.191749    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:54.191846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.191819    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:54.191895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.191849    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:34:54.262749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.262719    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" event={"ID":"fa525320-3c49-4e12-badd-234109092779","Type":"ContainerStarted","Data":"1194f52b3092704720774074b4d8d7ed7eb531177ebd20a52ef4645cab2dc38f"}
Oct 30 03:34:54.776104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.776076    4877 scope.go:122] "RemoveContainer" containerID="1a0cc079df00a6332556364fc9eb05a54d5392d8e130d0e74fde54fe48548998"
Oct 30 03:34:54.786063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.786044    4877 scope.go:122] "RemoveContainer" containerID="f3bcadf41176523ef4113ba0d1b803cdeab216d510bd242e053f89231a33caa2"
Oct 30 03:34:54.794810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.794790    4877 scope.go:122] "RemoveContainer" containerID="7df43414f03f6bfe75a688e1db1ef8b8589182f4506404a77b0057cfae176117"
Oct 30 03:34:54.803128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.803112    4877 scope.go:122] "RemoveContainer" containerID="c0bcd926818638e28f0ec60e586dd3b8645ad4b944b0ad4fdb44e759f6fb569c"
Oct 30 03:34:54.812053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.812036    4877 scope.go:122] "RemoveContainer" containerID="40f85f4f694054cbe4078719f358616bab57b2f44121e6c31d880c4256fd23c5"
Oct 30 03:34:54.820559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.820502    4877 scope.go:122] "RemoveContainer" containerID="e4198c6083ef4095d254296445ef1beb83f51a75465676cfd8eea3655f339002"
Oct 30 03:34:54.829080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.829062    4877 scope.go:122] "RemoveContainer" containerID="3dc6040855cdf83df67e57c7e742d6f8de8761d401adf6dab36e0e37ce38d2d5"
Oct 30 03:34:54.840577 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:54.839557    4877 scope.go:122] "RemoveContainer" containerID="7dd19f1e37efe2737f5e909bf1525b3c12e5fc9cbf46a40a81f4ed20545d2cda"
Oct 30 03:34:55.270614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:55.270516    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" event={"ID":"fa525320-3c49-4e12-badd-234109092779","Type":"ContainerStarted","Data":"ee8745bc46f50dc0f0b4ac14178aa552c3051b23aca475abed55cc38ea1a6f90"}
Oct 30 03:34:55.323600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:55.323539    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podStartSLOduration=2.323523799 podStartE2EDuration="2.323523799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:55.322218981 +0000 UTC m=+1923.751108399" watchObservedRunningTime="2026-10-30 03:34:55.323523799 +0000 UTC m=+1923.752413218"
Oct 30 03:34:55.861537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:55.861500    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:55.867298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:55.867274    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:56.277268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:56.277230    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:56.278237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:56.278216    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:34:57.208649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:57.208618    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 03:34:59.512408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.512353    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"]
Oct 30 03:34:59.519212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.519182    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.538512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.538478    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"]
Oct 30 03:34:59.545782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.545755    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7d99a4\""
Oct 30 03:34:59.545933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.545853    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7d99a4-dockercfg-t4bcm\""
Oct 30 03:34:59.577280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.577244    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"]
Oct 30 03:34:59.582984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.582958    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.600168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.600128    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"]
Oct 30 03:34:59.642446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.640164    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.642446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.640242    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbm8m\" (UniqueName: \"kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m\") pod \"payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.642446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.640276    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.642446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.640346    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktc2\" (UniqueName: \"kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2\") pod \"payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.741588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.741551    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.741780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.741606    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbm8m\" (UniqueName: \"kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m\") pod \"payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.741780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.741700    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.741780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.741749    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktc2\" (UniqueName: \"kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2\") pod \"payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.742460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.742398    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.742571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.742465    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.763501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.763437    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbm8m\" (UniqueName: \"kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m\") pod \"payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.765371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.765350    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktc2\" (UniqueName: \"kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2\") pod \"payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:34:59.834346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.834310    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:34:59.897658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:34:59.897630    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:35:00.154447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.154309    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"]
Oct 30 03:35:00.231123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.231092    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"]
Oct 30 03:35:00.233928 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:35:00.233903    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796823a0_9aee_4c59_be1c_e402d76b6cfe.slice/crio-21030dfcfb241219f656001947385ce662e78154f5b50d6f434fc0c8ab7b32aa WatchSource:0}: Error finding container 21030dfcfb241219f656001947385ce662e78154f5b50d6f434fc0c8ab7b32aa: Status 404 returned error can't find the container with id 21030dfcfb241219f656001947385ce662e78154f5b50d6f434fc0c8ab7b32aa
Oct 30 03:35:00.313948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.311288    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" event={"ID":"796823a0-9aee-4c59-be1c-e402d76b6cfe","Type":"ContainerStarted","Data":"21030dfcfb241219f656001947385ce662e78154f5b50d6f434fc0c8ab7b32aa"}
Oct 30 03:35:00.313948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.313601    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" event={"ID":"5f9908a7-9617-4e7f-98a3-7a83b0ddb569","Type":"ContainerStarted","Data":"4f497894e4b49cec3818712de391b3ef576c207b4ec19efb6e893673dbec20a8"}
Oct 30 03:35:00.313948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.313650    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" event={"ID":"5f9908a7-9617-4e7f-98a3-7a83b0ddb569","Type":"ContainerStarted","Data":"f0f59bb413d8393ecbc4fe5f4dd696ee9e5c6027aaf0d918603b83ddc4922159"}
Oct 30 03:35:00.316895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.314509    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:35:00.354335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:00.354279    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" podStartSLOduration=1.3542654650000001 podStartE2EDuration="1.354265465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:00.351104695 +0000 UTC m=+1928.779994114" watchObservedRunningTime="2026-10-30 03:35:00.354265465 +0000 UTC m=+1928.783154883"
Oct 30 03:35:01.337470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.336734    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" event={"ID":"796823a0-9aee-4c59-be1c-e402d76b6cfe","Type":"ContainerStarted","Data":"6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613"}
Oct 30 03:35:01.337470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.337204    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:35:01.856697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.856641    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" podStartSLOduration=2.856621321 podStartE2EDuration="2.856621321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:01.387855927 +0000 UTC m=+1929.816745346" watchObservedRunningTime="2026-10-30 03:35:01.856621321 +0000 UTC m=+1930.285510739"
Oct 30 03:35:01.859042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.859017    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"]
Oct 30 03:35:01.864185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.864163    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:01.895448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.893413    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6ad168-openshift-default-dockercfg-2jgqv\""
Oct 30 03:35:01.958331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.958264    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"]
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988039    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988129    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988180    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988308    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988352    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988377    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988402    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988455    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f6n\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.006466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:01.988501    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097091    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097164    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097191    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097217    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097249    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f6n\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097293    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097337    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.097447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097406    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.112013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.097471    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.112013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.098053    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.112013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.098487    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.112013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.098736    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.112013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.100311    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.112013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.106133    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.157493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.152123    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.161480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.157725    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.162963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.162915    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f6n\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.163955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.163932    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token\") pod \"e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") " pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.214447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.213684    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:02.649880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.649687    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"]
Oct 30 03:35:02.653762 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:35:02.653732    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07f1aad_625a_4cc5_b41c_6361b464df90.slice/crio-3ff6928c70e950b356fff022e27a73ba9c8066a4ea50366a17f2672c0d1665d2 WatchSource:0}: Error finding container 3ff6928c70e950b356fff022e27a73ba9c8066a4ea50366a17f2672c0d1665d2: Status 404 returned error can't find the container with id 3ff6928c70e950b356fff022e27a73ba9c8066a4ea50366a17f2672c0d1665d2
Oct 30 03:35:02.657103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.657065    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:02.657192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.657143    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:02.657192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:02.657172    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:03.355958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:03.355912    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" event={"ID":"a07f1aad-625a-4cc5-b41c-6361b464df90","Type":"ContainerStarted","Data":"aa321d45e0da2829935b3381269144f914b2d0ca8bf9e8f4f734c683e0786c91"}
Oct 30 03:35:03.356142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:03.355983    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" event={"ID":"a07f1aad-625a-4cc5-b41c-6361b464df90","Type":"ContainerStarted","Data":"3ff6928c70e950b356fff022e27a73ba9c8066a4ea50366a17f2672c0d1665d2"}
Oct 30 03:35:04.217448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:04.216436    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:04.236039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:04.235853    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:04.296598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:04.296510    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podStartSLOduration=3.296489618 podStartE2EDuration="3.296489618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:03.402418546 +0000 UTC m=+1931.831307967" watchObservedRunningTime="2026-10-30 03:35:04.296489618 +0000 UTC m=+1932.725379037"
Oct 30 03:35:04.364521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:04.364354    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:04.365638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:04.365621    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:07.975013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:07.974965    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"]
Oct 30 03:35:07.990975 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:07.990942    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.009899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.009871    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-547a78-openshift-default-dockercfg-qz5cs\""
Oct 30 03:35:08.011191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.011157    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"]
Oct 30 03:35:08.070264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070226    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070285    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070345    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070435    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070532    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgs7v\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070563    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070607    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070634    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.070879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.070725    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172131    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172181    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172211    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172232    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172250    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172320    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgs7v\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172365    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172416    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172472    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172738    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172752    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.172810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.172767    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.173044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.173019    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.173339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.173317    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.175556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.175530    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.175745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.175727    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.193520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.193484    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgs7v\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.194242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.194215    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.320479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.320439    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:08.676910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.676840    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"]
Oct 30 03:35:08.677038 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:35:08.676924    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a12af6_dfbf_4ee3_b76e_610e2c76f75f.slice/crio-a6f0cc7d9063ae1fa6803e446e4b5e3d35ecd3eb1310fb0786fbb3e05c515352 WatchSource:0}: Error finding container a6f0cc7d9063ae1fa6803e446e4b5e3d35ecd3eb1310fb0786fbb3e05c515352: Status 404 returned error can't find the container with id a6f0cc7d9063ae1fa6803e446e4b5e3d35ecd3eb1310fb0786fbb3e05c515352
Oct 30 03:35:08.679897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.679855    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:08.679992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.679946    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:08.680040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:08.679994    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:09.482819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:09.482781    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" event={"ID":"19a12af6-dfbf-4ee3-b76e-610e2c76f75f","Type":"ContainerStarted","Data":"f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"}
Oct 30 03:35:09.483327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:09.482838    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" event={"ID":"19a12af6-dfbf-4ee3-b76e-610e2c76f75f","Type":"ContainerStarted","Data":"a6f0cc7d9063ae1fa6803e446e4b5e3d35ecd3eb1310fb0786fbb3e05c515352"}
Oct 30 03:35:09.545337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:09.545267    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podStartSLOduration=2.545246723 podStartE2EDuration="2.545246723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:09.540628987 +0000 UTC m=+1937.969518423" watchObservedRunningTime="2026-10-30 03:35:09.545246723 +0000 UTC m=+1937.974136142"
Oct 30 03:35:10.321309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:10.321265    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:10.327092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:10.327066    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:10.491023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:10.490985    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:10.492235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:10.492205    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:11.152344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:11.152301    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:35:11.152548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:11.152379    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:35:11.192567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:11.192519    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:35:11.192763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:11.192570    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:35:12.350358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.350327    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 03:35:12.350933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.350400    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 03:35:12.507662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.507570    4877 generic.go:356] "Generic (PLEG): container finished" podID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerID="d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a" exitCode=0
Oct 30 03:35:12.507662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.507642    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" event={"ID":"ad6f100b-2891-446b-ab4d-5b7f535e9e00","Type":"ContainerDied","Data":"d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"}
Oct 30 03:35:12.508104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.508086    4877 scope.go:122] "RemoveContainer" containerID="d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"
Oct 30 03:35:12.509613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.509594    4877 generic.go:356] "Generic (PLEG): container finished" podID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerID="c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446" exitCode=0
Oct 30 03:35:12.509722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.509671    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" event={"ID":"97fa1ab1-f125-46d3-9836-9bc979fe9cd3","Type":"ContainerDied","Data":"c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"}
Oct 30 03:35:12.509989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.509960    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:12.510065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.510048    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:12.510125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.510096    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:12.510230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.510217    4877 scope.go:122] "RemoveContainer" containerID="c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"
Oct 30 03:35:12.511533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.511493    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:12.511651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.511578    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:12.511651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:12.511626    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:13.545617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.545576    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" event={"ID":"ad6f100b-2891-446b-ab4d-5b7f535e9e00","Type":"ContainerStarted","Data":"f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef"}
Oct 30 03:35:13.549235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.549198    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" event={"ID":"97fa1ab1-f125-46d3-9836-9bc979fe9cd3","Type":"ContainerStarted","Data":"557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91"}
Oct 30 03:35:13.642371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.642329    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"]
Oct 30 03:35:13.646994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.646970    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.664473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.664396    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"]
Oct 30 03:35:13.717830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.717792    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e1a0c2\""
Oct 30 03:35:13.718809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.718782    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e1a0c2-dockercfg-hpqrp\""
Oct 30 03:35:13.771153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.771107    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.771703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.771679    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5sqr\" (UniqueName: \"kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr\") pod \"payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.841507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.841050    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 03:35:13.841507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.841112    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 03:35:13.845446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.843022    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:35:13.872954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.872892    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5sqr\" (UniqueName: \"kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr\") pod \"payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.873172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.873036    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.874245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.874212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.938313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.938274    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5sqr\" (UniqueName: \"kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr\") pod \"payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:13.962527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:13.962493    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:14.262860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.262824    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:35:14.263115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.263092    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 03:35:14.263172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.263149    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 03:35:14.434482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.434445    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"]
Oct 30 03:35:14.559638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.559605    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" event={"ID":"51dd22c0-d892-4843-816a-71abd0d92556","Type":"ContainerStarted","Data":"fe520ee22381bad60d5a5e916c820a3d1a72683190091335526136bb387e6ef9"}
Oct 30 03:35:14.823447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.822835    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 03:35:14.823447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:14.822899    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 03:35:15.154850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.154772    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"]
Oct 30 03:35:15.159074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.159053    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.174135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.174108    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"]
Oct 30 03:35:15.253450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.216939    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-547a78-dockercfg-8r22p\""
Oct 30 03:35:15.308534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.220563    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-547a78\""
Oct 30 03:35:15.308795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.254382    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.312646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.312612    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxg6t\" (UniqueName: \"kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t\") pod \"payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.316445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.285673    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 03:35:15.316671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.316641    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 03:35:15.429926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.429828    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.431864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.431834    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxg6t\" (UniqueName: \"kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t\") pod \"payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.433095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.431205    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.452256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.452211    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxg6t\" (UniqueName: \"kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t\") pod \"payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.475173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.475145    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:15.571124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.571086    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" event={"ID":"51dd22c0-d892-4843-816a-71abd0d92556","Type":"ContainerStarted","Data":"f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19"}
Oct 30 03:35:15.571518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.571499    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:15.648702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.648636    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" podStartSLOduration=2.648620014 podStartE2EDuration="2.648620014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:15.630821218 +0000 UTC m=+1944.059710700" watchObservedRunningTime="2026-10-30 03:35:15.648620014 +0000 UTC m=+1944.077509433"
Oct 30 03:35:15.834459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.826010    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 03:35:15.834459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.826089    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 03:35:15.935610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:15.917546    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"]
Oct 30 03:35:16.262500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.262464    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 03:35:16.262703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.262520    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 03:35:16.270351 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:35:16.270317    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:35:16.596795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.596715    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" event={"ID":"6d74e3bf-7840-4a6c-a58b-1941d7a29da0","Type":"ContainerStarted","Data":"504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151"}
Oct 30 03:35:16.638449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.636708    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" event={"ID":"6d74e3bf-7840-4a6c-a58b-1941d7a29da0","Type":"ContainerStarted","Data":"2c973c5291e24141a9bfef8bc8ec2a75a074b5b6043ab9e29ed6d9796386cba7"}
Oct 30 03:35:16.638449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.636924    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:16.675857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.675808    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" podStartSLOduration=1.675794137 podStartE2EDuration="1.675794137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:16.672806119 +0000 UTC m=+1945.101695538" watchObservedRunningTime="2026-10-30 03:35:16.675794137 +0000 UTC m=+1945.104683555"
Oct 30 03:35:16.828448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.824907    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 03:35:16.828448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:16.824973    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 03:35:17.262699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:17.262661    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 03:35:17.262895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:17.262731    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 03:35:17.828878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:17.828840    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 03:35:17.829502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:17.829470    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 03:35:18.262679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:18.262595    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 03:35:18.262679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:18.262652    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 03:35:18.936835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:18.936786    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:35:18.938397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:18.938369    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:35:18.940705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:18.940612    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 03:35:19.266952 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:19.266875    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:35:19.267320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:19.267303    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:35:19.267956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:19.267939    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 03:35:25.042546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:25.042514    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"]
Oct 30 03:35:25.066253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:25.056329    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d5b356-tlh64"]
Oct 30 03:35:25.715573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:25.715533    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4b45fc-5d63-4b71-900f-ebb76e4800ee" path="/var/lib/kubelet/pods/8b4b45fc-5d63-4b71-900f-ebb76e4800ee/volumes"
Oct 30 03:35:26.618685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:26.618653    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 03:35:28.657397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:28.657367    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 03:35:31.783088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:31.783055    4877 generic.go:356] "Generic (PLEG): container finished" podID="fa525320-3c49-4e12-badd-234109092779" containerID="ee8745bc46f50dc0f0b4ac14178aa552c3051b23aca475abed55cc38ea1a6f90" exitCode=0
Oct 30 03:35:31.783559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:31.783133    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" event={"ID":"fa525320-3c49-4e12-badd-234109092779","Type":"ContainerDied","Data":"ee8745bc46f50dc0f0b4ac14178aa552c3051b23aca475abed55cc38ea1a6f90"}
Oct 30 03:35:31.783663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:31.783646    4877 scope.go:122] "RemoveContainer" containerID="ee8745bc46f50dc0f0b4ac14178aa552c3051b23aca475abed55cc38ea1a6f90"
Oct 30 03:35:31.785327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:31.785299    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:31.785407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:31.785374    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:31.785475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:31.785407    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:32.811866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:32.811822    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" event={"ID":"fa525320-3c49-4e12-badd-234109092779","Type":"ContainerStarted","Data":"1afb8d12561ffe1d7e1f7c70a4d168cfd2b7d3574a801da3565fe47c75f4be8e"}
Oct 30 03:35:32.860940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:32.860863    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:35:32.861170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:32.861145    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:35:32.861228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:32.861200    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 03:35:33.861331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:33.861295    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:35:33.861741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:33.861348    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 03:35:34.263151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.263069    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"]
Oct 30 03:35:34.268039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.268008    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.304543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.304511    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-852107-openshift-default-dockercfg-n7p5z\""
Oct 30 03:35:34.338075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.338042    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"]
Oct 30 03:35:34.402459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402180    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.402459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402252    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.402459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402309    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.402459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402346    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.402459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402387    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzq7\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.410765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402479    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.410765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402542    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.410765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402579    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.410765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.402631    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504416    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504500    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504564    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504710    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504767    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504815    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504859    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzq7\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.504936    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.513527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.505437    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.528228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.528192    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.536191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.529441    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.536191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.529688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.536475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.530175    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.536475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.535405    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.537058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.537008    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.547006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.546981    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzq7\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.550651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.550625    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token\") pod \"e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") " pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.634647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.634615    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:34.861179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.861146    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:35:34.861355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:34.861203    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 03:35:35.014371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.013864    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"]
Oct 30 03:35:35.015688 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.015649    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:35.015820 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.015732    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:35.015820 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.015775    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:35.838147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.838111    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" event={"ID":"cafd624a-1b4d-4710-a301-d6dff69bb92c","Type":"ContainerStarted","Data":"04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"}
Oct 30 03:35:35.838365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.838176    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" event={"ID":"cafd624a-1b4d-4710-a301-d6dff69bb92c","Type":"ContainerStarted","Data":"476bc90310706d2948fe36910bea0fcf6be6bc59b3a4b4cced9032e2f50eddfd"}
Oct 30 03:35:35.861629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.861595    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:35:35.862022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.861644    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 03:35:35.909018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:35.908945    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podStartSLOduration=1.9089236779999998 podStartE2EDuration="1.908923678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:35.896692249 +0000 UTC m=+1964.325581706" watchObservedRunningTime="2026-10-30 03:35:35.908923678 +0000 UTC m=+1964.337813097"
Oct 30 03:35:36.635846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:36.635811    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:36.641951 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:36.641925    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:36.846863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:36.846795    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:36.847769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:36.847751    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:35:36.861322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:36.861291    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 03:35:36.861497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:36.861339    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 03:35:37.915197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:37.915159    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:35:37.915750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:37.915717    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:35:37.917288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:37.917265    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 03:35:42.333315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.333262    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"]
Oct 30 03:35:42.341307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.341281    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.356505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.356479    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-817531-openshift-default-dockercfg-6ld4j\""
Oct 30 03:35:42.363594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.363547    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.363948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.363909    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.364151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.364123    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.364291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.364273    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.364463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.364446    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.364739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.364713    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.364908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.364879    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skdw\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.365062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.365035    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.365269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.365214    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.432460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.429803    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"]
Oct 30 03:35:42.466964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.466912    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.467268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.467243    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.467498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.467479    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.467656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.467641    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.467902 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.467872    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.468050 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.468032    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skdw\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.468162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.468148    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.468321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.468293    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.468540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.468507    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.470324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.470265    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.471912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.470751    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.472069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.470968    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.472178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.471593    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.472280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.471826    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.529889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.529853    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.539305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.539235    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.585827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.578874    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skdw\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.655448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.655367    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token\") pod \"e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") " pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:42.682956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:42.682684    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:43.136857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:43.136826    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"]
Oct 30 03:35:43.141616 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:35:43.140040    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58abc3ac_e0c7_451d_8b8b_2f6e39ea89af.slice/crio-267b505f7e7b4898fd822e74577a6610c81f081bff2cc07cc9661b47188dab6c WatchSource:0}: Error finding container 267b505f7e7b4898fd822e74577a6610c81f081bff2cc07cc9661b47188dab6c: Status 404 returned error can't find the container with id 267b505f7e7b4898fd822e74577a6610c81f081bff2cc07cc9661b47188dab6c
Oct 30 03:35:43.142980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:43.142948    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:43.143053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:43.143014    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:43.143053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:43.143044    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:43.963071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:43.963030    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" event={"ID":"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af","Type":"ContainerStarted","Data":"7ababe62034b7444763eb5f84f31411c767c61646d2a96d348dc19e308d487c2"}
Oct 30 03:35:43.963560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:43.963115    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" event={"ID":"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af","Type":"ContainerStarted","Data":"267b505f7e7b4898fd822e74577a6610c81f081bff2cc07cc9661b47188dab6c"}
Oct 30 03:35:44.039629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:44.039456    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podStartSLOduration=2.039438834 podStartE2EDuration="2.039438834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:44.037171059 +0000 UTC m=+1972.466060476" watchObservedRunningTime="2026-10-30 03:35:44.039438834 +0000 UTC m=+1972.468328257"
Oct 30 03:35:44.695446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:44.694817    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:44.717449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:44.710684    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:44.971024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:44.970935    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:44.972064 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:44.972044    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:35:46.229067 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:35:46.228971    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:35:48.852757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.852714    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"]
Oct 30 03:35:48.893828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.893794    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"]
Oct 30 03:35:48.894541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.894519    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:48.922403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.922374    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-817531\""
Oct 30 03:35:48.922684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.922667    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-817531-dockercfg-qh4g8\""
Oct 30 03:35:48.985657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.985577    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:48.985657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:48.985638    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dms\" (UniqueName: \"kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms\") pod \"payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:49.087529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:49.087417    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:49.087529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:49.087533    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dms\" (UniqueName: \"kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms\") pod \"payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:49.088312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:49.088288    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:49.106648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:49.106575    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dms\" (UniqueName: \"kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms\") pod \"payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:49.244407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:49.244366    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:49.631542 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:35:49.630537    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d38f70_a370_4375_8264_8e4e55a446bf.slice/crio-51755c4948f800dcbd96f72e7ca9045b597a67d1bb808432cd3c71ee10b37a09 WatchSource:0}: Error finding container 51755c4948f800dcbd96f72e7ca9045b597a67d1bb808432cd3c71ee10b37a09: Status 404 returned error can't find the container with id 51755c4948f800dcbd96f72e7ca9045b597a67d1bb808432cd3c71ee10b37a09
Oct 30 03:35:49.631756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:49.631603    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"]
Oct 30 03:35:50.048037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:50.047998    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" event={"ID":"00d38f70-a370-4375-8264-8e4e55a446bf","Type":"ContainerStarted","Data":"2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15"}
Oct 30 03:35:50.696944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:50.048067    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" event={"ID":"00d38f70-a370-4375-8264-8e4e55a446bf","Type":"ContainerStarted","Data":"51755c4948f800dcbd96f72e7ca9045b597a67d1bb808432cd3c71ee10b37a09"}
Oct 30 03:35:50.696944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:50.048098    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:35:50.696944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:50.089780    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" podStartSLOduration=2.089760978 podStartE2EDuration="2.089760978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:50.085488806 +0000 UTC m=+1978.514378219" watchObservedRunningTime="2026-10-30 03:35:50.089760978 +0000 UTC m=+1978.518650399"
Oct 30 03:35:50.696944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:50.407535    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:35:50.696944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:50.407589    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:35:51.056512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.056479    4877 generic.go:356] "Generic (PLEG): container finished" podID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerID="aa321d45e0da2829935b3381269144f914b2d0ca8bf9e8f4f734c683e0786c91" exitCode=0
Oct 30 03:35:51.057020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.056559    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" event={"ID":"a07f1aad-625a-4cc5-b41c-6361b464df90","Type":"ContainerDied","Data":"aa321d45e0da2829935b3381269144f914b2d0ca8bf9e8f4f734c683e0786c91"}
Oct 30 03:35:51.057104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.057069    4877 scope.go:122] "RemoveContainer" containerID="aa321d45e0da2829935b3381269144f914b2d0ca8bf9e8f4f734c683e0786c91"
Oct 30 03:35:51.058583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.058560    4877 generic.go:356] "Generic (PLEG): container finished" podID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerID="f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259" exitCode=0
Oct 30 03:35:51.058679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.058627    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" event={"ID":"19a12af6-dfbf-4ee3-b76e-610e2c76f75f","Type":"ContainerDied","Data":"f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"}
Oct 30 03:35:51.058761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.058737    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:51.058818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.058805    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:51.058854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.058846    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:51.059139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.059122    4877 scope.go:122] "RemoveContainer" containerID="f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"
Oct 30 03:35:51.060358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.060335    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:51.060439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.060392    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:51.060485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:51.060436    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:35:52.068517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.068471    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" event={"ID":"19a12af6-dfbf-4ee3-b76e-610e2c76f75f","Type":"ContainerStarted","Data":"3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5"}
Oct 30 03:35:52.070811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.070787    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" event={"ID":"a07f1aad-625a-4cc5-b41c-6361b464df90","Type":"ContainerStarted","Data":"483a7d4e81846b6a74ea991ba3246266879a607a699318071cedfb2b4ec59919"}
Oct 30 03:35:52.228453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.227550    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:35:52.228453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.227610    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:35:52.228453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.227888    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:52.326507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.325837    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:52.339482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.336557    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 03:35:52.339482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:52.336624    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 03:35:53.217943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:53.217899    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:35:53.218616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:53.218582    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:35:53.323849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:53.323811    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 03:35:53.324138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:53.324110    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 03:35:54.214930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:54.214887    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:35:54.215241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:54.215206    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:35:54.325909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:54.325867    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 03:35:54.326552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:54.326519    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 03:35:55.216958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:55.216918    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:35:55.217299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:55.217251    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:35:55.325595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:55.325540    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 03:35:55.325828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:55.325633    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 03:35:55.492414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:55.492318    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:56.222506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:56.219855    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:35:56.222506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:56.219918    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:35:56.321878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:56.321834    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 03:35:56.322183 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:56.322154    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 03:35:57.216151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:57.216116    4877 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 03:35:57.216802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:57.216751    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 03:35:57.321854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:57.321816    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Oct 30 03:35:57.322164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:57.322137    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Oct 30 03:35:58.220160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:58.220125    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:58.220599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:58.220532    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:58.221323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:58.221300    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 03:35:58.329383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:58.329349    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:35:58.331270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:35:58.331245    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 03:36:01.062253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:01.062214    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 03:36:16.230614 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:36:16.230572    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:36:16.585229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.585193    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"]
Oct 30 03:36:16.590332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.590303    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.618867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.618834    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"openshift-service-ca.crt\""
Oct 30 03:36:16.619900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.619878    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"kube-root-ca.crt\""
Oct 30 03:36:16.627027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.627000    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"default-dockercfg-v8mlq\""
Oct 30 03:36:16.627154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.627080    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"auth-test-model-e1a0c2-kserve-self-signed-certs\""
Oct 30 03:36:16.627303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.627273    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"]
Oct 30 03:36:16.643312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.643272    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.643534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.643325    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngbg\" (UniqueName: \"kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.643534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.643496    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.643678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.643623    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.643678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.643661    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.643758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.643725    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745144    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngbg\" (UniqueName: \"kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745210    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745275    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745305    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745353    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745512    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745715    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745812    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.745892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.745871    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.747774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.747751    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.747936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.747918    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.763486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.763461    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngbg\" (UniqueName: \"kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg\") pod \"auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:16.935098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:16.935006    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:17.246624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:17.246595    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"]
Oct 30 03:36:17.248013 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:36:17.247985    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b660c2_50ce_484a_ba48_53dc330bbc02.slice/crio-738f9bd292a340847f1daef5dc3eb1695386ecb5ef511761d6cbf04b040a3b43 WatchSource:0}: Error finding container 738f9bd292a340847f1daef5dc3eb1695386ecb5ef511761d6cbf04b040a3b43: Status 404 returned error can't find the container with id 738f9bd292a340847f1daef5dc3eb1695386ecb5ef511761d6cbf04b040a3b43
Oct 30 03:36:17.321325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:17.321295    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m" event={"ID":"f3b660c2-50ce-484a-ba48-53dc330bbc02","Type":"ContainerStarted","Data":"709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0"}
Oct 30 03:36:17.321480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:17.321340    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m" event={"ID":"f3b660c2-50ce-484a-ba48-53dc330bbc02","Type":"ContainerStarted","Data":"738f9bd292a340847f1daef5dc3eb1695386ecb5ef511761d6cbf04b040a3b43"}
Oct 30 03:36:17.361530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:17.361468    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m" podStartSLOduration=1.361450005 podStartE2EDuration="1.361450005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:17.358498632 +0000 UTC m=+2005.787388051" watchObservedRunningTime="2026-10-30 03:36:17.361450005 +0000 UTC m=+2005.790339423"
Oct 30 03:36:17.936076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:17.936038    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:26.935508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:26.935410    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:26.953081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:26.953052    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:27.485479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:27.485395    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:36:29.508732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:29.508688    4877 generic.go:356] "Generic (PLEG): container finished" podID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerID="04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469" exitCode=0
Oct 30 03:36:29.509329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:29.508777    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" event={"ID":"cafd624a-1b4d-4710-a301-d6dff69bb92c","Type":"ContainerDied","Data":"04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"}
Oct 30 03:36:29.509475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:29.509407    4877 scope.go:122] "RemoveContainer" containerID="04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"
Oct 30 03:36:29.511531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:29.511503    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:36:29.511642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:29.511593    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:36:29.511707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:29.511639    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:36:30.519442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:30.519377    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" event={"ID":"cafd624a-1b4d-4710-a301-d6dff69bb92c","Type":"ContainerStarted","Data":"55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9"}
Oct 30 03:36:30.635395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:30.635359    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:36:30.635684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:30.635556    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:36:30.635839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:30.635818    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:36:31.637308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:31.636874    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:36:31.637308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:31.636936    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:36:32.636862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:32.636820    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:36:32.637203 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:32.637170    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:36:33.637037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:33.636829    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:36:33.637605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:33.637565    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:36:34.636252 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:34.636212    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:36:34.636456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:34.636269    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:36:35.635432 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:35.635396    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Oct 30 03:36:35.635942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:35.635482    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Oct 30 03:36:36.642380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:36.642348    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:36:36.642801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:36.642756    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:36:36.643593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:36.643572    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 03:36:38.602046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:38.602006    4877 generic.go:356] "Generic (PLEG): container finished" podID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerID="7ababe62034b7444763eb5f84f31411c767c61646d2a96d348dc19e308d487c2" exitCode=0
Oct 30 03:36:38.602613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:38.602080    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" event={"ID":"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af","Type":"ContainerDied","Data":"7ababe62034b7444763eb5f84f31411c767c61646d2a96d348dc19e308d487c2"}
Oct 30 03:36:38.602690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:38.602612    4877 scope.go:122] "RemoveContainer" containerID="7ababe62034b7444763eb5f84f31411c767c61646d2a96d348dc19e308d487c2"
Oct 30 03:36:38.604183 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:38.604156    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:36:38.604261 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:38.604225    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:36:38.604299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:38.604258    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:36:39.618166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:39.618124    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" event={"ID":"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af","Type":"ContainerStarted","Data":"53aecb49627b3294e058397fdbe7291cf11faa26dd47688d7cbb20313a34ba22"}
Oct 30 03:36:39.685226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:39.684786    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:36:39.685226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:39.684913    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:36:39.685226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:39.684953    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 03:36:40.696446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:40.685774    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:36:40.696446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:40.685835    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 03:36:41.683796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:41.683755    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:36:41.684106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:41.684077    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 03:36:42.693459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:42.688808    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:36:42.693459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:42.688872    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 03:36:43.689791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:43.689749    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:36:43.694570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:43.694534    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 03:36:44.684184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:44.684142    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 03:36:44.684370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:44.684217    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 03:36:44.971405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:44.971325    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:36:45.688818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:45.688767    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:36:45.690074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:45.690044    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 03:36:46.229811 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:36:46.229772    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:36:54.949228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:36:54.949196    4877 scope.go:122] "RemoveContainer" containerID="762f3cdc5c6f4e45192e7e5e322d7101d302bac131038c55391647301b6caafa"
Oct 30 03:37:16.228873 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:37:16.228835    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:37:22.501373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.501330    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"]
Oct 30 03:37:22.507054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.507030    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.531056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.531029    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"]
Oct 30 03:37:22.538454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.537668    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-817531\"/\"openshift-service-ca.crt\""
Oct 30 03:37:22.538454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.537704    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-817531\"/\"default-dockercfg-dgxzw\""
Oct 30 03:37:22.538454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.537708    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-817531\"/\"kube-root-ca.crt\""
Oct 30 03:37:22.538454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.537775    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-817531\"/\"auth-test-model-817531-kserve-self-signed-certs\""
Oct 30 03:37:22.564245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.564219    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.564379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.564260    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz486\" (UniqueName: \"kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.564379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.564332    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.564511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.564391    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.564511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.564500    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.564595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.564528    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.665997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.665966    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.665997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666000    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666089    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666125    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz486\" (UniqueName: \"kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666166    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666220    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666442    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666572    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.666666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.666647    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.668223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.668203    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.669100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.669079    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.684546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.684525    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz486\" (UniqueName: \"kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486\") pod \"auth-test-model-817531-kserve-864c5567bf-s2298\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") " pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:22.824787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:22.824733    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:23.140072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:23.140043    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"]
Oct 30 03:37:23.140531 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:37:23.140504    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd59a9592_12c4_465a_bc9c_81ca22ff3118.slice/crio-81e5d0de994001ded29de1c33c399bd2f762f09a67ab1d116990cf4a62b9bfa3 WatchSource:0}: Error finding container 81e5d0de994001ded29de1c33c399bd2f762f09a67ab1d116990cf4a62b9bfa3: Status 404 returned error can't find the container with id 81e5d0de994001ded29de1c33c399bd2f762f09a67ab1d116990cf4a62b9bfa3
Oct 30 03:37:23.977458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:23.977378    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298" event={"ID":"d59a9592-12c4-465a-bc9c-81ca22ff3118","Type":"ContainerStarted","Data":"f0aeb1fe1c252ec6ce67e1a744606ecf4eb1852859090ab7897ed08859ea76e8"}
Oct 30 03:37:23.977458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:23.977458    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298" event={"ID":"d59a9592-12c4-465a-bc9c-81ca22ff3118","Type":"ContainerStarted","Data":"81e5d0de994001ded29de1c33c399bd2f762f09a67ab1d116990cf4a62b9bfa3"}
Oct 30 03:37:24.018379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:24.018323    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298" podStartSLOduration=2.018309806 podStartE2EDuration="2.018309806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:24.015326553 +0000 UTC m=+2072.444215996" watchObservedRunningTime="2026-10-30 03:37:24.018309806 +0000 UTC m=+2072.447199224"
Oct 30 03:37:24.825154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:24.825114    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:32.825403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:32.825364    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:32.838267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:32.838242    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:33.056937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:33.056903    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:37:34.251405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.251367    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"]
Oct 30 03:37:34.255832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.255810    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.271602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.271569    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-75c2b0-openshift-default-dockercfg-68zgd\""
Oct 30 03:37:34.288279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.288238    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"]
Oct 30 03:37:34.432611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.432574    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.432919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.432879    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.432937    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.432999    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.433025    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwtd\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.433062    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.433151    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.433232    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.433440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.433301    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.534781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.534752    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535418    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535830    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535857    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535281    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535917    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535944    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwtd\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.535984 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.535981    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.536269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.536075    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.536269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.536158    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.536269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.536257    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.536675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.536650    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.536839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.536817    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.536979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.536954    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.539374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.539307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.539646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.539627    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.554963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.554932    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwtd\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.555104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.555053    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token\") pod \"e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") " pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.571471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.571439    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:34.909565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.909535    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"]
Oct 30 03:37:34.910512 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:37:34.910488    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da40340_f601_43f2_b10d_c3b09c2bad0f.slice/crio-e461a6df42d4903fc08884e43065545c8fa6ac7f5285dcd6ba7026ab1794e2b3 WatchSource:0}: Error finding container e461a6df42d4903fc08884e43065545c8fa6ac7f5285dcd6ba7026ab1794e2b3: Status 404 returned error can't find the container with id e461a6df42d4903fc08884e43065545c8fa6ac7f5285dcd6ba7026ab1794e2b3
Oct 30 03:37:34.913396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.913363    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:37:34.913520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.913454    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:37:34.913520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:34.913502    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:37:35.065333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:35.065291    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" event={"ID":"8da40340-f601-43f2-b10d-c3b09c2bad0f","Type":"ContainerStarted","Data":"67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"}
Oct 30 03:37:35.066619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:35.066593    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" event={"ID":"8da40340-f601-43f2-b10d-c3b09c2bad0f","Type":"ContainerStarted","Data":"e461a6df42d4903fc08884e43065545c8fa6ac7f5285dcd6ba7026ab1794e2b3"}
Oct 30 03:37:35.112301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:35.112249    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podStartSLOduration=1.112233168 podStartE2EDuration="1.112233168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:35.107750302 +0000 UTC m=+2083.536639720" watchObservedRunningTime="2026-10-30 03:37:35.112233168 +0000 UTC m=+2083.541122586"
Oct 30 03:37:35.574446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:35.572865    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:36.578920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:36.578890    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:37.079608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:37.079569    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:37.080693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:37.080671    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:37:39.292322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.292284    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"]
Oct 30 03:37:39.297344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.297324    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.313754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.313711    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume\") pod \"payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") " pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.313754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.313766    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjt66\" (UniqueName: \"kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66\") pod \"payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") " pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.325246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.325217    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-75c2b0\""
Oct 30 03:37:39.331660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.331633    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"]
Oct 30 03:37:39.539448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.536089    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume\") pod \"payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") " pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.539448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.536144    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjt66\" (UniqueName: \"kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66\") pod \"payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") " pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.539448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.537651    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume\") pod \"payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") " pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.555942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.555871    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjt66\" (UniqueName: \"kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66\") pod \"payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") " pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.612199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.612163    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:39.932656 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:37:39.932622    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb03a4f_2d67_4b27_a719_5f03b8b90f56.slice/crio-d6e5b066facb8dbd5c22a8d125df0208da4ebf4d2c7a298048581c7307d58e0e WatchSource:0}: Error finding container d6e5b066facb8dbd5c22a8d125df0208da4ebf4d2c7a298048581c7307d58e0e: Status 404 returned error can't find the container with id d6e5b066facb8dbd5c22a8d125df0208da4ebf4d2c7a298048581c7307d58e0e
Oct 30 03:37:39.933159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:39.933134    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"]
Oct 30 03:37:40.108139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:40.108092    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9" event={"ID":"0eb03a4f-2d67-4b27-a719-5f03b8b90f56","Type":"ContainerStarted","Data":"46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a"}
Oct 30 03:37:40.108331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:40.108166    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:40.108331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:40.108187    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9" event={"ID":"0eb03a4f-2d67-4b27-a719-5f03b8b90f56","Type":"ContainerStarted","Data":"d6e5b066facb8dbd5c22a8d125df0208da4ebf4d2c7a298048581c7307d58e0e"}
Oct 30 03:37:40.150478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:40.150405    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9" podStartSLOduration=1.150391507 podStartE2EDuration="1.150391507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:40.148413126 +0000 UTC m=+2088.577302550" watchObservedRunningTime="2026-10-30 03:37:40.150391507 +0000 UTC m=+2088.579280934"
Oct 30 03:37:46.235798 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:37:46.235754    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:37:51.120764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:51.120730    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:37:52.244212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:52.244175    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:37:52.266908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:37:52.266881    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:38:16.234837 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:38:16.234759    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:38:23.120628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:23.120579    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:38:23.121054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:23.120690    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:38:24.507955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:24.507916    4877 generic.go:356] "Generic (PLEG): container finished" podID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerID="67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27" exitCode=0
Oct 30 03:38:24.508502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:24.507968    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" event={"ID":"8da40340-f601-43f2-b10d-c3b09c2bad0f","Type":"ContainerDied","Data":"67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"}
Oct 30 03:38:24.508502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:24.508462    4877 scope.go:122] "RemoveContainer" containerID="67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"
Oct 30 03:38:24.510062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:24.510025    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:38:24.510168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:24.510114    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:38:24.510168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:24.510146    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:38:25.520765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:25.520733    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" event={"ID":"8da40340-f601-43f2-b10d-c3b09c2bad0f","Type":"ContainerStarted","Data":"042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83"}
Oct 30 03:38:25.571806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:25.571774    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:38:25.572070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:25.572041    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 03:38:25.572157 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:25.572103    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 03:38:26.571814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:26.571778    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 03:38:26.572218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:26.571856    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 03:38:27.572491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:27.572456    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 03:38:27.572895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:27.572519    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 03:38:28.572670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:28.572636    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 03:38:28.573084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:28.572699    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 03:38:29.572159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:29.572116    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 03:38:29.572365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:29.572172    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 03:38:30.572454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:30.572402    4877 patch_prober.go:29] interesting pod/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 03:38:30.572871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:30.572481    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 03:38:31.576582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:31.576547    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:38:31.577067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:31.576940    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:38:31.577627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:31.577607    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:38:41.930811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:41.930767    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"]
Oct 30 03:38:41.935372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:41.935354    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:41.953626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:41.953600    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"]
Oct 30 03:38:41.960053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:41.960026    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:38:41.960181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:41.960062    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:38:42.077587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.077553    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.077777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.077753    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjmv\" (UniqueName: \"kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.077881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.077866    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.179078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.179045    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjmv\" (UniqueName: \"kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.179262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.179121    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.179262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.179211    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.179869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.179840    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.181700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.181640    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.197867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.197833    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjmv\" (UniqueName: \"kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv\") pod \"maas-api-revoke-sub-52120ea8-krcc8\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.251146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.251112    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:42.560161 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:38:42.560128    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c66b214_00e3_4863_94a2_4c12e0341221.slice/crio-b305735e430e020a349d569c4e885dd5b4c80b395f144a0cea202e1b7087d2db WatchSource:0}: Error finding container b305735e430e020a349d569c4e885dd5b4c80b395f144a0cea202e1b7087d2db: Status 404 returned error can't find the container with id b305735e430e020a349d569c4e885dd5b4c80b395f144a0cea202e1b7087d2db
Oct 30 03:38:42.560579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.560555    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"]
Oct 30 03:38:42.721295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:42.721258    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8" event={"ID":"0c66b214-00e3-4863-94a2-4c12e0341221","Type":"ContainerStarted","Data":"b305735e430e020a349d569c4e885dd5b4c80b395f144a0cea202e1b7087d2db"}
Oct 30 03:38:43.730370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:43.730331    4877 generic.go:356] "Generic (PLEG): container finished" podID="0c66b214-00e3-4863-94a2-4c12e0341221" containerID="9afbf8bf7a3c629a4dd95ee284646e0c5ae3832b2e2c6b7b6682a73eb198ca56" exitCode=0
Oct 30 03:38:43.730803 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:43.730413    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8" event={"ID":"0c66b214-00e3-4863-94a2-4c12e0341221","Type":"ContainerDied","Data":"9afbf8bf7a3c629a4dd95ee284646e0c5ae3832b2e2c6b7b6682a73eb198ca56"}
Oct 30 03:38:44.909970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:44.909940    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:45.019834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.019738    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv\" (UniqueName: \"kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv\") pod \"0c66b214-00e3-4863-94a2-4c12e0341221\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") "
Oct 30 03:38:45.019834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.019794    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca\") pod \"0c66b214-00e3-4863-94a2-4c12e0341221\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") "
Oct 30 03:38:45.020075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.020019    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca\") pod \"0c66b214-00e3-4863-94a2-4c12e0341221\" (UID: \"0c66b214-00e3-4863-94a2-4c12e0341221\") "
Oct 30 03:38:45.020463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.020396    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca" pod "0c66b214-00e3-4863-94a2-4c12e0341221" (UID: "0c66b214-00e3-4863-94a2-4c12e0341221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:45.020608 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.020576    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:38:45.023414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.023384    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca" pod "0c66b214-00e3-4863-94a2-4c12e0341221" (UID: "0c66b214-00e3-4863-94a2-4c12e0341221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:45.024315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.024290    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv" pod "0c66b214-00e3-4863-94a2-4c12e0341221" (UID: "0c66b214-00e3-4863-94a2-4c12e0341221"). InnerVolumeSpecName "kube-api-access-rhjmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:38:45.121768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.121735    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c66b214-00e3-4863-94a2-4c12e0341221-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:38:45.121768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.121765    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhjmv\" (UniqueName: \"kubernetes.io/projected/0c66b214-00e3-4863-94a2-4c12e0341221-kube-api-access-rhjmv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:38:45.746462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.746407    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8" event={"ID":"0c66b214-00e3-4863-94a2-4c12e0341221","Type":"ContainerDied","Data":"b305735e430e020a349d569c4e885dd5b4c80b395f144a0cea202e1b7087d2db"}
Oct 30 03:38:45.746944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.746483    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b305735e430e020a349d569c4e885dd5b4c80b395f144a0cea202e1b7087d2db"
Oct 30 03:38:45.746944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:45.746437    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"
Oct 30 03:38:46.233598 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:38:46.233564    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:38:52.277498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.277462    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"]
Oct 30 03:38:52.278169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.278152    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0c66b214-00e3-4863-94a2-4c12e0341221" containerName="revoke-keys"
Oct 30 03:38:52.278218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.278172    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c66b214-00e3-4863-94a2-4c12e0341221" containerName="revoke-keys"
Oct 30 03:38:52.278284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.278274    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c66b214-00e3-4863-94a2-4c12e0341221" containerName="revoke-keys"
Oct 30 03:38:52.281903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.281884    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.296968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.296936    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:38:52.302535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.302510    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"]
Oct 30 03:38:52.308873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.308851    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:38:52.310643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.310618    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.310808 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.310778    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfjh\" (UniqueName: \"kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.310908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.310890    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.412276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.412239    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.412525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.412310    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfjh\" (UniqueName: \"kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.412525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.412361    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.413037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.413010    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.414703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.414680    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.431916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.431885    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfjh\" (UniqueName: \"kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh\") pod \"maas-api-revoke-sub-68a12324-kpsv5\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.597151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.597068    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:52.914379 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:38:52.914348    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb67331_e1f7_4def_ba47_f4abd71c21d3.slice/crio-27cd3cdb55bb1b827a1edfec973f0f643fa0a0159b76e4b5d91509c616a884a6 WatchSource:0}: Error finding container 27cd3cdb55bb1b827a1edfec973f0f643fa0a0159b76e4b5d91509c616a884a6: Status 404 returned error can't find the container with id 27cd3cdb55bb1b827a1edfec973f0f643fa0a0159b76e4b5d91509c616a884a6
Oct 30 03:38:52.914686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:52.914661    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"]
Oct 30 03:38:53.813605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:53.813564    4877 generic.go:356] "Generic (PLEG): container finished" podID="6fb67331-e1f7-4def-ba47-f4abd71c21d3" containerID="b5f74ee87a088a4eafa6ce89bcfa44a91401adcadc36b259460ecfffc4c28676" exitCode=0
Oct 30 03:38:53.814003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:53.813638    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5" event={"ID":"6fb67331-e1f7-4def-ba47-f4abd71c21d3","Type":"ContainerDied","Data":"b5f74ee87a088a4eafa6ce89bcfa44a91401adcadc36b259460ecfffc4c28676"}
Oct 30 03:38:53.814003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:53.813700    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5" event={"ID":"6fb67331-e1f7-4def-ba47-f4abd71c21d3","Type":"ContainerStarted","Data":"27cd3cdb55bb1b827a1edfec973f0f643fa0a0159b76e4b5d91509c616a884a6"}
Oct 30 03:38:54.987053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:54.987023    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:55.049363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.049318    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca\") pod \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") "
Oct 30 03:38:55.049572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.049507    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh\" (UniqueName: \"kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh\") pod \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") "
Oct 30 03:38:55.049662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.049623    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca\") pod \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\" (UID: \"6fb67331-e1f7-4def-ba47-f4abd71c21d3\") "
Oct 30 03:38:55.049771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.049725    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca" pod "6fb67331-e1f7-4def-ba47-f4abd71c21d3" (UID: "6fb67331-e1f7-4def-ba47-f4abd71c21d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:55.050083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.050049    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:38:55.051818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.051789    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca" pod "6fb67331-e1f7-4def-ba47-f4abd71c21d3" (UID: "6fb67331-e1f7-4def-ba47-f4abd71c21d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:55.051906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.051819    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh" pod "6fb67331-e1f7-4def-ba47-f4abd71c21d3" (UID: "6fb67331-e1f7-4def-ba47-f4abd71c21d3"). InnerVolumeSpecName "kube-api-access-5cfjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:38:55.151845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.151769    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cfjh\" (UniqueName: \"kubernetes.io/projected/6fb67331-e1f7-4def-ba47-f4abd71c21d3-kube-api-access-5cfjh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:38:55.151845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.151797    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb67331-e1f7-4def-ba47-f4abd71c21d3-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:38:55.832114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.832086    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"
Oct 30 03:38:55.832283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.832078    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5" event={"ID":"6fb67331-e1f7-4def-ba47-f4abd71c21d3","Type":"ContainerDied","Data":"27cd3cdb55bb1b827a1edfec973f0f643fa0a0159b76e4b5d91509c616a884a6"}
Oct 30 03:38:55.832283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:38:55.832210    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27cd3cdb55bb1b827a1edfec973f0f643fa0a0159b76e4b5d91509c616a884a6"
Oct 30 03:39:06.309955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.309911    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"]
Oct 30 03:39:06.310598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.310581    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6fb67331-e1f7-4def-ba47-f4abd71c21d3" containerName="revoke-keys"
Oct 30 03:39:06.310598 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.310597    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb67331-e1f7-4def-ba47-f4abd71c21d3" containerName="revoke-keys"
Oct 30 03:39:06.310739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.310705    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb67331-e1f7-4def-ba47-f4abd71c21d3" containerName="revoke-keys"
Oct 30 03:39:06.314541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.314519    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.332615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.332587    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"]
Oct 30 03:39:06.341456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.339958    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:39:06.341456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.340085    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:39:06.416209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.416172    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.416402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.416320    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2xq\" (UniqueName: \"kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.416509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.416452    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.517619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.517579    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.517826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.517687    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2xq\" (UniqueName: \"kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.517826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.517810    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.518323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.518303    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.520203 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.520181    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.536754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.536728    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2xq\" (UniqueName: \"kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq\") pod \"maas-api-revoke-sub-15ec39aa-vhhrl\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.631196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.631111    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:06.937251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.937223    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"]
Oct 30 03:39:06.939466 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:06.939418    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9871fe07_f664_4d0f_ad8f_9fbb54d6f3a3.slice/crio-6a6de4d9fcbe96bde1e95e76cc9b45393446bc4319eec6313e999734faf92142 WatchSource:0}: Error finding container 6a6de4d9fcbe96bde1e95e76cc9b45393446bc4319eec6313e999734faf92142: Status 404 returned error can't find the container with id 6a6de4d9fcbe96bde1e95e76cc9b45393446bc4319eec6313e999734faf92142
Oct 30 03:39:06.980586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:06.980553    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl" event={"ID":"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3","Type":"ContainerStarted","Data":"6a6de4d9fcbe96bde1e95e76cc9b45393446bc4319eec6313e999734faf92142"}
Oct 30 03:39:07.993060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:07.993025    4877 generic.go:356] "Generic (PLEG): container finished" podID="9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" containerID="0a3e25735a19e657874efbcb84787ed4c9b5296d71a658ae756b6afe44353119" exitCode=0
Oct 30 03:39:07.993541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:07.993100    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl" event={"ID":"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3","Type":"ContainerDied","Data":"0a3e25735a19e657874efbcb84787ed4c9b5296d71a658ae756b6afe44353119"}
Oct 30 03:39:08.070414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:08.070375    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"]
Oct 30 03:39:09.173068 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.173039    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:09.255980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.255897    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca\") pod \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") "
Oct 30 03:39:09.256162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.255995    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq\" (UniqueName: \"kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq\") pod \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") "
Oct 30 03:39:09.256162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.256033    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca\") pod \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\" (UID: \"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3\") "
Oct 30 03:39:09.256521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.256492    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca" pod "9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" (UID: "9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:09.258046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.258024    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca" pod "9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" (UID: "9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:09.258241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.258227    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq" pod "9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" (UID: "9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3"). InnerVolumeSpecName "kube-api-access-tm2xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:09.357693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.357657    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:09.357693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.357687    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:09.357693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:09.357697    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm2xq\" (UniqueName: \"kubernetes.io/projected/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3-kube-api-access-tm2xq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:10.012163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:10.012088    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"
Oct 30 03:39:10.012325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:10.012079    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl" event={"ID":"9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3","Type":"ContainerDied","Data":"6a6de4d9fcbe96bde1e95e76cc9b45393446bc4319eec6313e999734faf92142"}
Oct 30 03:39:10.012325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:10.012209    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a6de4d9fcbe96bde1e95e76cc9b45393446bc4319eec6313e999734faf92142"
Oct 30 03:39:14.798674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:14.798629    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"]
Oct 30 03:39:16.229613 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:39:16.229570    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:39:16.717798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.717757    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"]
Oct 30 03:39:16.718740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.718720    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" containerName="revoke-keys"
Oct 30 03:39:16.718807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.718743    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" containerName="revoke-keys"
Oct 30 03:39:16.718881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.718871    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" containerName="revoke-keys"
Oct 30 03:39:16.722402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.722384    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.736606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.736579    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:39:16.739864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.739840    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"]
Oct 30 03:39:16.746715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.746691    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:39:16.772630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.772596    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.772793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.772737    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.772854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.772817    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlpd\" (UniqueName: \"kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.874490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.874450    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.874663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.874588    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlpd\" (UniqueName: \"kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.874720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.874703    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.875371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.875345    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.876856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.876834    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:16.892946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:16.892917    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlpd\" (UniqueName: \"kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd\") pod \"maas-api-revoke-sub-df0acca6-5psn9\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:17.036714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:17.036681    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:17.344042 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:17.344006    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba0babe_489f_4ba7_89ca_4aaf347a32cc.slice/crio-25e0daceb80446d22506f007362493cf3dd9a5c583fece145be9e8dd8682bd72 WatchSource:0}: Error finding container 25e0daceb80446d22506f007362493cf3dd9a5c583fece145be9e8dd8682bd72: Status 404 returned error can't find the container with id 25e0daceb80446d22506f007362493cf3dd9a5c583fece145be9e8dd8682bd72
Oct 30 03:39:17.344379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:17.344210    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"]
Oct 30 03:39:18.085741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:18.085708    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9" event={"ID":"fba0babe-489f-4ba7-89ca-4aaf347a32cc","Type":"ContainerStarted","Data":"51bce21728bac0d7a8f5b7e2d3fadf71a4cb332015f7d74b500e434c9f2e3767"}
Oct 30 03:39:18.085902 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:18.085759    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9" event={"ID":"fba0babe-489f-4ba7-89ca-4aaf347a32cc","Type":"ContainerStarted","Data":"25e0daceb80446d22506f007362493cf3dd9a5c583fece145be9e8dd8682bd72"}
Oct 30 03:39:18.127403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:18.127331    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9" podStartSLOduration=2.127311866 podStartE2EDuration="2.127311866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:18.124228797 +0000 UTC m=+2186.553118254" watchObservedRunningTime="2026-10-30 03:39:18.127311866 +0000 UTC m=+2186.556201285"
Oct 30 03:39:19.094973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:19.094939    4877 generic.go:356] "Generic (PLEG): container finished" podID="fba0babe-489f-4ba7-89ca-4aaf347a32cc" containerID="51bce21728bac0d7a8f5b7e2d3fadf71a4cb332015f7d74b500e434c9f2e3767" exitCode=0
Oct 30 03:39:19.095342 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:19.095019    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9" event={"ID":"fba0babe-489f-4ba7-89ca-4aaf347a32cc","Type":"ContainerDied","Data":"51bce21728bac0d7a8f5b7e2d3fadf71a4cb332015f7d74b500e434c9f2e3767"}
Oct 30 03:39:19.799846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:19.799781    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9" podUID="0eb03a4f-2d67-4b27-a719-5f03b8b90f56" containerName="payload-processing" containerID="cri-o://46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a" gracePeriod=25
Oct 30 03:39:20.094687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.094662    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:39:20.104595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.104558    4877 generic.go:356] "Generic (PLEG): container finished" podID="0eb03a4f-2d67-4b27-a719-5f03b8b90f56" containerID="46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a" exitCode=0
Oct 30 03:39:20.104988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.104612    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"
Oct 30 03:39:20.104988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.104642    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9" event={"ID":"0eb03a4f-2d67-4b27-a719-5f03b8b90f56","Type":"ContainerDied","Data":"46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a"}
Oct 30 03:39:20.104988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.104698    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9" event={"ID":"0eb03a4f-2d67-4b27-a719-5f03b8b90f56","Type":"ContainerDied","Data":"d6e5b066facb8dbd5c22a8d125df0208da4ebf4d2c7a298048581c7307d58e0e"}
Oct 30 03:39:20.104988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.104727    4877 scope.go:122] "RemoveContainer" containerID="46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a"
Oct 30 03:39:20.117847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.117825    4877 scope.go:122] "RemoveContainer" containerID="46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a"
Oct 30 03:39:20.118179 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:39:20.118156    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a\": container with ID starting with 46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a not found: ID does not exist" containerID="46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a"
Oct 30 03:39:20.118302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.118186    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a"} err="failed to get container status \"46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a\": rpc error: code = NotFound desc = could not find container \"46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a\": container with ID starting with 46826489206c2f3ae83b5f912226a922a07b06cb3de0a2603fc473eb6d01ca5a not found: ID does not exist"
Oct 30 03:39:20.217349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.217308    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume\") pod \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") "
Oct 30 03:39:20.217764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.217735    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66\" (UniqueName: \"kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66\") pod \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\" (UID: \"0eb03a4f-2d67-4b27-a719-5f03b8b90f56\") "
Oct 30 03:39:20.219855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.219810    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume" pod "0eb03a4f-2d67-4b27-a719-5f03b8b90f56" (UID: "0eb03a4f-2d67-4b27-a719-5f03b8b90f56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:20.220642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.220614    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66" pod "0eb03a4f-2d67-4b27-a719-5f03b8b90f56" (UID: "0eb03a4f-2d67-4b27-a719-5f03b8b90f56"). InnerVolumeSpecName "kube-api-access-kjt66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:20.287277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.287252    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:20.319327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.319246    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd\" (UniqueName: \"kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd\") pod \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") "
Oct 30 03:39:20.319327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.319309    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca\") pod \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") "
Oct 30 03:39:20.319562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.319543    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca\") pod \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\" (UID: \"fba0babe-489f-4ba7-89ca-4aaf347a32cc\") "
Oct 30 03:39:20.319770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.319745    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca" pod "fba0babe-489f-4ba7-89ca-4aaf347a32cc" (UID: "fba0babe-489f-4ba7-89ca-4aaf347a32cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:20.320161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.320119    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:20.320161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.320152    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:20.320345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.320169    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjt66\" (UniqueName: \"kubernetes.io/projected/0eb03a4f-2d67-4b27-a719-5f03b8b90f56-kube-api-access-kjt66\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:20.321674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.321645    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca" pod "fba0babe-489f-4ba7-89ca-4aaf347a32cc" (UID: "fba0babe-489f-4ba7-89ca-4aaf347a32cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:20.321749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.321705    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd" pod "fba0babe-489f-4ba7-89ca-4aaf347a32cc" (UID: "fba0babe-489f-4ba7-89ca-4aaf347a32cc"). InnerVolumeSpecName "kube-api-access-pxlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:20.421581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.421549    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba0babe-489f-4ba7-89ca-4aaf347a32cc-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:20.421581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.421578    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlpd\" (UniqueName: \"kubernetes.io/projected/fba0babe-489f-4ba7-89ca-4aaf347a32cc-kube-api-access-pxlpd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:20.454144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.454109    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"]
Oct 30 03:39:20.457769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:20.457743    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75c2b0-69456d9fc6-g95m9"]
Oct 30 03:39:21.115169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:21.115087    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"
Oct 30 03:39:21.115169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:21.115124    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9" event={"ID":"fba0babe-489f-4ba7-89ca-4aaf347a32cc","Type":"ContainerDied","Data":"25e0daceb80446d22506f007362493cf3dd9a5c583fece145be9e8dd8682bd72"}
Oct 30 03:39:21.115694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:21.115183    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e0daceb80446d22506f007362493cf3dd9a5c583fece145be9e8dd8682bd72"
Oct 30 03:39:21.702631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:21.702598    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb03a4f-2d67-4b27-a719-5f03b8b90f56" path="/var/lib/kubelet/pods/0eb03a4f-2d67-4b27-a719-5f03b8b90f56/volumes"
Oct 30 03:39:22.207591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.207554    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"]
Oct 30 03:39:22.208595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.208574    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fba0babe-489f-4ba7-89ca-4aaf347a32cc" containerName="revoke-keys"
Oct 30 03:39:22.208662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.208598    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba0babe-489f-4ba7-89ca-4aaf347a32cc" containerName="revoke-keys"
Oct 30 03:39:22.208700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.208668    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0eb03a4f-2d67-4b27-a719-5f03b8b90f56" containerName="payload-processing"
Oct 30 03:39:22.208700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.208677    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb03a4f-2d67-4b27-a719-5f03b8b90f56" containerName="payload-processing"
Oct 30 03:39:22.208852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.208840    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb03a4f-2d67-4b27-a719-5f03b8b90f56" containerName="payload-processing"
Oct 30 03:39:22.208904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.208862    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba0babe-489f-4ba7-89ca-4aaf347a32cc" containerName="revoke-keys"
Oct 30 03:39:22.213335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.213310    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.233130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.233102    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"]
Oct 30 03:39:22.240016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.239979    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-689bcf-openshift-default-dockercfg-rngkt\""
Oct 30 03:39:22.349737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.349689    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.349955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.349757    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.349955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.349823    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f84s\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.349955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.349873    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.349955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.349906    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.350195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.349962    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.350195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.350086    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.350195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.350119    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.350345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.350267    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.452802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.452755    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f84s\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.453231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453204    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.453377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453361    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.453562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453539    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.453799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453782    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.453923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453909    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454022 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453993    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.454207    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.453962    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.454511    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.454645    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.454803    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.454942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.454925    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.455443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.455404    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.457982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.457926    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.458624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.458603    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.471868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.471840    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f84s\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.472291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.472272    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token\") pod \"e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") " pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.527956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.527920    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:22.865124 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:22.865076    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1b44f8_0fcb_4f16_85e1_87698900b711.slice/crio-a24d8c8fcd394f9c0de0b5378ff2b3bc273c733cab22cbb8011986b3f6702dae WatchSource:0}: Error finding container a24d8c8fcd394f9c0de0b5378ff2b3bc273c733cab22cbb8011986b3f6702dae: Status 404 returned error can't find the container with id a24d8c8fcd394f9c0de0b5378ff2b3bc273c733cab22cbb8011986b3f6702dae
Oct 30 03:39:22.865327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.865218    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"]
Oct 30 03:39:22.867740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.867707    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:39:22.867833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.867773    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:39:22.867833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:22.867823    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:39:23.030913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.030878    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"]
Oct 30 03:39:23.036985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.036946    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d09dbda-kc9lk"]
Oct 30 03:39:23.093171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.093122    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m" podUID="f3b660c2-50ce-484a-ba48-53dc330bbc02" containerName="main" containerID="cri-o://709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0" gracePeriod=45
Oct 30 03:39:23.135564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.135480    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" event={"ID":"dd1b44f8-0fcb-4f16-85e1-87698900b711","Type":"ContainerStarted","Data":"7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"}
Oct 30 03:39:23.135564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.135554    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" event={"ID":"dd1b44f8-0fcb-4f16-85e1-87698900b711","Type":"ContainerStarted","Data":"a24d8c8fcd394f9c0de0b5378ff2b3bc273c733cab22cbb8011986b3f6702dae"}
Oct 30 03:39:23.178138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.178078    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podStartSLOduration=1.178062728 podStartE2EDuration="1.178062728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:23.174722983 +0000 UTC m=+2191.603612421" watchObservedRunningTime="2026-10-30 03:39:23.178062728 +0000 UTC m=+2191.606952145"
Oct 30 03:39:23.390530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.390462    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:39:23.483769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.483733    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs\") pod \"f3b660c2-50ce-484a-ba48-53dc330bbc02\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") "
Oct 30 03:39:23.483930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.483802    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg\" (UniqueName: \"kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg\") pod \"f3b660c2-50ce-484a-ba48-53dc330bbc02\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") "
Oct 30 03:39:23.483930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.483893    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir\") pod \"f3b660c2-50ce-484a-ba48-53dc330bbc02\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") "
Oct 30 03:39:23.484047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.484011    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home\") pod \"f3b660c2-50ce-484a-ba48-53dc330bbc02\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") "
Oct 30 03:39:23.484236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.484204    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir" pod "f3b660c2-50ce-484a-ba48-53dc330bbc02" (UID: "f3b660c2-50ce-484a-ba48-53dc330bbc02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:23.484236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.484226    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home" pod "f3b660c2-50ce-484a-ba48-53dc330bbc02" (UID: "f3b660c2-50ce-484a-ba48-53dc330bbc02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:23.484361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.484307    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache\") pod \"f3b660c2-50ce-484a-ba48-53dc330bbc02\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") "
Oct 30 03:39:23.484481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.484466    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm\") pod \"f3b660c2-50ce-484a-ba48-53dc330bbc02\" (UID: \"f3b660c2-50ce-484a-ba48-53dc330bbc02\") "
Oct 30 03:39:23.484640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.484611    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache" pod "f3b660c2-50ce-484a-ba48-53dc330bbc02" (UID: "f3b660c2-50ce-484a-ba48-53dc330bbc02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:23.485058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.485033    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.485058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.485060    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.485229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.485073    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.486782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.486757    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs" pod "f3b660c2-50ce-484a-ba48-53dc330bbc02" (UID: "f3b660c2-50ce-484a-ba48-53dc330bbc02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:23.486866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.486828    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm" pod "f3b660c2-50ce-484a-ba48-53dc330bbc02" (UID: "f3b660c2-50ce-484a-ba48-53dc330bbc02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:23.486866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.486830    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg" pod "f3b660c2-50ce-484a-ba48-53dc330bbc02" (UID: "f3b660c2-50ce-484a-ba48-53dc330bbc02"). InnerVolumeSpecName "kube-api-access-9ngbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:23.528779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.528743    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:23.534853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.534827    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:23.586503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.586470    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3b660c2-50ce-484a-ba48-53dc330bbc02-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.586503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.586505    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b660c2-50ce-484a-ba48-53dc330bbc02-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.586718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.586520    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ngbg\" (UniqueName: \"kubernetes.io/projected/f3b660c2-50ce-484a-ba48-53dc330bbc02-kube-api-access-9ngbg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:23.699564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:23.699486    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f813ab01-ee7f-4519-9237-7b641466dc3c" path="/var/lib/kubelet/pods/f813ab01-ee7f-4519-9237-7b641466dc3c/volumes"
Oct 30 03:39:24.144585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.144541    4877 generic.go:356] "Generic (PLEG): container finished" podID="f3b660c2-50ce-484a-ba48-53dc330bbc02" containerID="709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0" exitCode=0
Oct 30 03:39:24.144768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.144595    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"
Oct 30 03:39:24.144768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.144629    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m" event={"ID":"f3b660c2-50ce-484a-ba48-53dc330bbc02","Type":"ContainerDied","Data":"709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0"}
Oct 30 03:39:24.144768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.144680    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m" event={"ID":"f3b660c2-50ce-484a-ba48-53dc330bbc02","Type":"ContainerDied","Data":"738f9bd292a340847f1daef5dc3eb1695386ecb5ef511761d6cbf04b040a3b43"}
Oct 30 03:39:24.144768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.144701    4877 scope.go:122] "RemoveContainer" containerID="709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0"
Oct 30 03:39:24.145282 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.145159    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:24.146286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.146268    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:39:24.155893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.155871    4877 scope.go:122] "RemoveContainer" containerID="709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0"
Oct 30 03:39:24.156220 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:39:24.156201    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0\": container with ID starting with 709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0 not found: ID does not exist" containerID="709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0"
Oct 30 03:39:24.156299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.156226    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0"} err="failed to get container status \"709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0\": rpc error: code = NotFound desc = could not find container \"709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0\": container with ID starting with 709f98439a2405a41657bcd63f9604daf7169debab58acb2f96f4ea3fe02cff0 not found: ID does not exist"
Oct 30 03:39:24.198279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.198244    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"]
Oct 30 03:39:24.208622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:24.208591    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/auth-test-model-e1a0c2-kserve-84cd7fdf65-jgs8m"]
Oct 30 03:39:25.701769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:25.701728    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b660c2-50ce-484a-ba48-53dc330bbc02" path="/var/lib/kubelet/pods/f3b660c2-50ce-484a-ba48-53dc330bbc02/volumes"
Oct 30 03:39:27.426378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.426338    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"]
Oct 30 03:39:27.427338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.427310    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f3b660c2-50ce-484a-ba48-53dc330bbc02" containerName="main"
Oct 30 03:39:27.427338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.427332    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b660c2-50ce-484a-ba48-53dc330bbc02" containerName="main"
Oct 30 03:39:27.427572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.427561    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b660c2-50ce-484a-ba48-53dc330bbc02" containerName="main"
Oct 30 03:39:27.432487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.432469    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.449047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.449022    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"]
Oct 30 03:39:27.455547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.455521    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-689bcf\""
Oct 30 03:39:27.471138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.471115    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-689bcf-dockercfg-m8cjv\""
Oct 30 03:39:27.548444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.547331    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.548444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.547518    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2n9h\" (UniqueName: \"kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h\") pod \"payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.606604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.602314    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"]
Oct 30 03:39:27.627807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.627775    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.646141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.644698    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"]
Oct 30 03:39:27.650180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.648469    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n9h\" (UniqueName: \"kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h\") pod \"payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.650180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.648615    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.650180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.649392    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.668417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.667213    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2n9h\" (UniqueName: \"kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h\") pod \"payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.746240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.746150    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:27.749410 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.749271    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.749540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.749515    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg87l\" (UniqueName: \"kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l\") pod \"payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.851272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.851236    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg87l\" (UniqueName: \"kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l\") pod \"payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.851534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.851369    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.852093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.852071    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.870310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.870273    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg87l\" (UniqueName: \"kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l\") pod \"payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:27.954502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:27.954467    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:28.058344 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:28.058313    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7a9816_26a9_46ca_b754_493cf1e031d8.slice/crio-bffc8f9145c858bab3d25c82bd734f1db8949d9da7d1f9a2f47f851e80f7995a WatchSource:0}: Error finding container bffc8f9145c858bab3d25c82bd734f1db8949d9da7d1f9a2f47f851e80f7995a: Status 404 returned error can't find the container with id bffc8f9145c858bab3d25c82bd734f1db8949d9da7d1f9a2f47f851e80f7995a
Oct 30 03:39:28.058541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:28.058521    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"]
Oct 30 03:39:28.176745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:28.176704    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" event={"ID":"bd7a9816-26a9-46ca-b754-493cf1e031d8","Type":"ContainerStarted","Data":"fd65c96a5c4fd99438bf356d4587ce1b99e32c64e513e9c26a49b854bc795142"}
Oct 30 03:39:28.176928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:28.176763    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" event={"ID":"bd7a9816-26a9-46ca-b754-493cf1e031d8","Type":"ContainerStarted","Data":"bffc8f9145c858bab3d25c82bd734f1db8949d9da7d1f9a2f47f851e80f7995a"}
Oct 30 03:39:28.176928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:28.176802    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:28.216995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:28.216932    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" podStartSLOduration=1.216915649 podStartE2EDuration="1.216915649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:28.215950848 +0000 UTC m=+2196.644840266" watchObservedRunningTime="2026-10-30 03:39:28.216915649 +0000 UTC m=+2196.645805067"
Oct 30 03:39:28.271504 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:28.267855    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad02199_2cdc_4e9a_98cf_8200baf66843.slice/crio-95f8d0e3822ea21b8aa87c3e49ac5b4f791135756f7f9d421a31508da8a28622 WatchSource:0}: Error finding container 95f8d0e3822ea21b8aa87c3e49ac5b4f791135756f7f9d421a31508da8a28622: Status 404 returned error can't find the container with id 95f8d0e3822ea21b8aa87c3e49ac5b4f791135756f7f9d421a31508da8a28622
Oct 30 03:39:28.271504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:28.269703    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"]
Oct 30 03:39:29.186691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:29.186647    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" event={"ID":"4ad02199-2cdc-4e9a-98cf-8200baf66843","Type":"ContainerStarted","Data":"c030d509fd8717ffdba4ec749475b890d9e66d4e53ace1fdedc535e28b3571b0"}
Oct 30 03:39:29.187198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:29.186721    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" event={"ID":"4ad02199-2cdc-4e9a-98cf-8200baf66843","Type":"ContainerStarted","Data":"95f8d0e3822ea21b8aa87c3e49ac5b4f791135756f7f9d421a31508da8a28622"}
Oct 30 03:39:29.187198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:29.186967    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:29.237663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:29.237558    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" podStartSLOduration=2.237544326 podStartE2EDuration="2.237544326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:29.23531187 +0000 UTC m=+2197.664201287" watchObservedRunningTime="2026-10-30 03:39:29.237544326 +0000 UTC m=+2197.666433753"
Oct 30 03:39:37.067292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.067187    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"]
Oct 30 03:39:37.067735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.067483    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy" containerID="cri-o://042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83" gracePeriod=30
Oct 30 03:39:37.119055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.119015    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"]
Oct 30 03:39:37.125306 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.125276    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.157827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.157765    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9d8692-openshift-default-dockercfg-n6mwr\""
Oct 30 03:39:37.164318 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.164287    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"]
Oct 30 03:39:37.307029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.306985    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307314 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307275    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307349    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307469    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307534    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307560    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307648    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307724    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjtf\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.307814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.307785    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.408987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.408895    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.408987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.408956    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409013    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409101    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409156    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409183    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409282    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409352    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409359    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjtf\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409467    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409519    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409557    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.409985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.409948    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.410145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.410110    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.413418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.413389    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.414360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.414338    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.438076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.432902    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjtf\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.438076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.435214    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token\") pod \"e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") " pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.449881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.449854    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:37.781586 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:37.781549    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300ee1d9_af02_4b23_93e4_d90143518f26.slice/crio-cf37b87934e4e76d423fbebd5a0b179d3010e217a23e8f40360c42d08ce6ecfb WatchSource:0}: Error finding container cf37b87934e4e76d423fbebd5a0b179d3010e217a23e8f40360c42d08ce6ecfb: Status 404 returned error can't find the container with id cf37b87934e4e76d423fbebd5a0b179d3010e217a23e8f40360c42d08ce6ecfb
Oct 30 03:39:37.782670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.782645    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"]
Oct 30 03:39:37.784412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.784378    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:39:37.784505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.784458    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:39:37.784505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:37.784497    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:39:38.364447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:38.362623    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" event={"ID":"300ee1d9-af02-4b23-93e4-d90143518f26","Type":"ContainerStarted","Data":"ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"}
Oct 30 03:39:38.364447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:38.362689    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" event={"ID":"300ee1d9-af02-4b23-93e4-d90143518f26","Type":"ContainerStarted","Data":"cf37b87934e4e76d423fbebd5a0b179d3010e217a23e8f40360c42d08ce6ecfb"}
Oct 30 03:39:38.450562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:38.450536    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:38.456581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:38.456551    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:38.499800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:38.499750    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podStartSLOduration=1.499734579 podStartE2EDuration="1.499734579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:38.410304424 +0000 UTC m=+2206.839193892" watchObservedRunningTime="2026-10-30 03:39:38.499734579 +0000 UTC m=+2206.928623997"
Oct 30 03:39:39.189476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:39.189444    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 03:39:39.371007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:39.370977    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:39.372130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:39.372109    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:39:40.197873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:40.197843    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 03:39:42.358707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.358679    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:39:42.390950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.390917    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391008    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391079    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391117    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391150    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391192    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:42.391300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391217    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391332    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391400    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391447    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs\") pod \"8da40340-f601-43f2-b10d-c3b09c2bad0f\" (UID: \"8da40340-f601-43f2-b10d-c3b09c2bad0f\") "
Oct 30 03:39:42.391674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391590    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:42.391738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391692    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:42.392199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391949    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:42.392199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391975    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.392199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.391992    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.392199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.392005    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da40340-f601-43f2-b10d-c3b09c2bad0f-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.392199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.392170    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:42.395163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.395033    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:42.395163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.395110    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:39:42.395163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.395129    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "kube-api-access-4gwtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:42.395387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.395286    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo" pod "8da40340-f601-43f2-b10d-c3b09c2bad0f" (UID: "8da40340-f601-43f2-b10d-c3b09c2bad0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:39:42.399584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.399560    4877 generic.go:356] "Generic (PLEG): container finished" podID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerID="042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83" exitCode=0
Oct 30 03:39:42.399784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.399766    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" event={"ID":"8da40340-f601-43f2-b10d-c3b09c2bad0f","Type":"ContainerDied","Data":"042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83"}
Oct 30 03:39:42.399855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.399812    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr" event={"ID":"8da40340-f601-43f2-b10d-c3b09c2bad0f","Type":"ContainerDied","Data":"e461a6df42d4903fc08884e43065545c8fa6ac7f5285dcd6ba7026ab1794e2b3"}
Oct 30 03:39:42.399855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.399834    4877 scope.go:122] "RemoveContainer" containerID="042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83"
Oct 30 03:39:42.399985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.399946    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"
Oct 30 03:39:42.443967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.443942    4877 scope.go:122] "RemoveContainer" containerID="67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"
Oct 30 03:39:42.457205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.457182    4877 scope.go:122] "RemoveContainer" containerID="042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83"
Oct 30 03:39:42.457625 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:39:42.457603    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83\": container with ID starting with 042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83 not found: ID does not exist" containerID="042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83"
Oct 30 03:39:42.457738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.457632    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83"} err="failed to get container status \"042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83\": rpc error: code = NotFound desc = could not find container \"042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83\": container with ID starting with 042a0e7fdbda9d7686ca0e8565d8119d7244d37bdf4e5150d5c9705b6e349b83 not found: ID does not exist"
Oct 30 03:39:42.457738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.457651    4877 scope.go:122] "RemoveContainer" containerID="67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"
Oct 30 03:39:42.457985 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:39:42.457938    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27\": container with ID starting with 67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27 not found: ID does not exist" containerID="67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"
Oct 30 03:39:42.457985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.457960    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27"} err="failed to get container status \"67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27\": rpc error: code = NotFound desc = could not find container \"67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27\": container with ID starting with 67d3a7b1cbfdd9b35de77242aa9cc15d4642480254022b310cc8ed6f301ddf27 not found: ID does not exist"
Oct 30 03:39:42.487896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.487866    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"]
Oct 30 03:39:42.491657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.491630    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-75c2b0-openshift-default-65fffb95d5-s6svr"]
Oct 30 03:39:42.493102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.493082    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gwtd\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-kube-api-access-4gwtd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.493174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.493107    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.493174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.493121    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.493174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.493134    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.493174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.493146    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:42.493174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:42.493157    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da40340-f601-43f2-b10d-c3b09c2bad0f-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:43.712714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:43.708875    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" path="/var/lib/kubelet/pods/8da40340-f601-43f2-b10d-c3b09c2bad0f/volumes"
Oct 30 03:39:46.229802 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:39:46.229765    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:39:55.879811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.879768    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"]
Oct 30 03:39:55.880459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.880444    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy"
Oct 30 03:39:55.880459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.880459    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy"
Oct 30 03:39:55.880548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.880500    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy"
Oct 30 03:39:55.880548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.880506    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy"
Oct 30 03:39:55.880627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.880617    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy"
Oct 30 03:39:55.885518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.885496    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:55.903321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.903294    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"]
Oct 30 03:39:55.909932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.909904    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:39:55.910057 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:55.910045    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:39:56.011589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.011555    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.011780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.011750    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.011840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.011815    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4dk\" (UniqueName: \"kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.113233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.113195    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.113475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.113417    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.113570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.113552    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4dk\" (UniqueName: \"kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.114052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.114022    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.115859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.115831    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.132078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.132026    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4dk\" (UniqueName: \"kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk\") pod \"maas-api-revoke-sub-47467683-lwljc\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.201164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.201134    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:56.510531 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:56.510497    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cfcdea_e95b_4f48_9cdb_4f24eb57df0d.slice/crio-9fe55cfb554ae3deb016297f89c7bcc421363b9976696509e04bcba65326f205 WatchSource:0}: Error finding container 9fe55cfb554ae3deb016297f89c7bcc421363b9976696509e04bcba65326f205: Status 404 returned error can't find the container with id 9fe55cfb554ae3deb016297f89c7bcc421363b9976696509e04bcba65326f205
Oct 30 03:39:56.512131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.512089    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"]
Oct 30 03:39:56.526599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:56.526573    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc" event={"ID":"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d","Type":"ContainerStarted","Data":"9fe55cfb554ae3deb016297f89c7bcc421363b9976696509e04bcba65326f205"}
Oct 30 03:39:57.305532 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.305496    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"]
Oct 30 03:39:57.306288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.306271    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da40340-f601-43f2-b10d-c3b09c2bad0f" containerName="istio-proxy"
Oct 30 03:39:57.309954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.309931    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.330439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.330398    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"]
Oct 30 03:39:57.432153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.432124    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.432338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.432238    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8djn\" (UniqueName: \"kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.432338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.432332    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.533510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.533472    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.533716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.533598    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8djn\" (UniqueName: \"kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.533716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.533671    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.534298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.534273    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.535611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.535584    4877 generic.go:356] "Generic (PLEG): container finished" podID="21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" containerID="4f18360192367a251c448a262eb8466ee6c4575a9273bc1cc8ddea6b19d06f90" exitCode=0
Oct 30 03:39:57.535711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.535675    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc" event={"ID":"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d","Type":"ContainerDied","Data":"4f18360192367a251c448a262eb8466ee6c4575a9273bc1cc8ddea6b19d06f90"}
Oct 30 03:39:57.536231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.536213    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.552181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.552154    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8djn\" (UniqueName: \"kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn\") pod \"maas-api-revoke-sub-0f58b94a-9gnnw\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.644909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.644823    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:39:57.960839 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:39:57.960809    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b9418b_70fc_42c7_8b72_3ca70d3f08b2.slice/crio-0d95e0a4cc8afe8399be4f013bb428059c6955b0b0819cb8da662b296730c2fa WatchSource:0}: Error finding container 0d95e0a4cc8afe8399be4f013bb428059c6955b0b0819cb8da662b296730c2fa: Status 404 returned error can't find the container with id 0d95e0a4cc8afe8399be4f013bb428059c6955b0b0819cb8da662b296730c2fa
Oct 30 03:39:57.961724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:57.961701    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"]
Oct 30 03:39:58.544699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.544654    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw" event={"ID":"47b9418b-70fc-42c7-8b72-3ca70d3f08b2","Type":"ContainerStarted","Data":"bafee640f55e21707cb0a8abee29b8afc6c13117243d831e6159b870209248a0"}
Oct 30 03:39:58.545118 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.544762    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw" event={"ID":"47b9418b-70fc-42c7-8b72-3ca70d3f08b2","Type":"ContainerStarted","Data":"0d95e0a4cc8afe8399be4f013bb428059c6955b0b0819cb8da662b296730c2fa"}
Oct 30 03:39:58.586389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.586289    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw" podStartSLOduration=1.586274671 podStartE2EDuration="1.586274671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:58.583003246 +0000 UTC m=+2227.011892665" watchObservedRunningTime="2026-10-30 03:39:58.586274671 +0000 UTC m=+2227.015164089"
Oct 30 03:39:58.717570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.717538    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:58.853224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.853183    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca\") pod \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") "
Oct 30 03:39:58.853475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.853338    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca\") pod \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") "
Oct 30 03:39:58.853475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.853387    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk\" (UniqueName: \"kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk\") pod \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\" (UID: \"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d\") "
Oct 30 03:39:58.853707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.853662    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca" pod "21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" (UID: "21cfcdea-e95b-4f48-9cdb-4f24eb57df0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:39:58.853970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.853948    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:58.855641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.855608    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca" pod "21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" (UID: "21cfcdea-e95b-4f48-9cdb-4f24eb57df0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:39:58.855726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.855664    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk" pod "21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" (UID: "21cfcdea-e95b-4f48-9cdb-4f24eb57df0d"). InnerVolumeSpecName "kube-api-access-2v4dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:39:58.955679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.955593    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:58.955679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:58.955623    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4dk\" (UniqueName: \"kubernetes.io/projected/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d-kube-api-access-2v4dk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:39:59.554111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:59.554073    4877 generic.go:356] "Generic (PLEG): container finished" podID="47b9418b-70fc-42c7-8b72-3ca70d3f08b2" containerID="bafee640f55e21707cb0a8abee29b8afc6c13117243d831e6159b870209248a0" exitCode=0
Oct 30 03:39:59.554560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:59.554154    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw" event={"ID":"47b9418b-70fc-42c7-8b72-3ca70d3f08b2","Type":"ContainerDied","Data":"bafee640f55e21707cb0a8abee29b8afc6c13117243d831e6159b870209248a0"}
Oct 30 03:39:59.555699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:59.555675    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc" event={"ID":"21cfcdea-e95b-4f48-9cdb-4f24eb57df0d","Type":"ContainerDied","Data":"9fe55cfb554ae3deb016297f89c7bcc421363b9976696509e04bcba65326f205"}
Oct 30 03:39:59.555833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:59.555714    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"
Oct 30 03:39:59.555833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:39:59.555721    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe55cfb554ae3deb016297f89c7bcc421363b9976696509e04bcba65326f205"
Oct 30 03:40:00.965583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:00.965550    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:40:01.027693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.027655    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca\") pod \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") "
Oct 30 03:40:01.027906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.027781    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn\" (UniqueName: \"kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn\") pod \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") "
Oct 30 03:40:01.027906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.027825    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca\") pod \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\" (UID: \"47b9418b-70fc-42c7-8b72-3ca70d3f08b2\") "
Oct 30 03:40:01.028766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.028735    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca" pod "47b9418b-70fc-42c7-8b72-3ca70d3f08b2" (UID: "47b9418b-70fc-42c7-8b72-3ca70d3f08b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:01.035977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.035934    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca" pod "47b9418b-70fc-42c7-8b72-3ca70d3f08b2" (UID: "47b9418b-70fc-42c7-8b72-3ca70d3f08b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:01.039200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.039158    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn" pod "47b9418b-70fc-42c7-8b72-3ca70d3f08b2" (UID: "47b9418b-70fc-42c7-8b72-3ca70d3f08b2"). InnerVolumeSpecName "kube-api-access-h8djn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:01.129932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.129832    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.129932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.129873    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8djn\" (UniqueName: \"kubernetes.io/projected/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-kube-api-access-h8djn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.129932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.129887    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47b9418b-70fc-42c7-8b72-3ca70d3f08b2-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:01.584604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.584561    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw" event={"ID":"47b9418b-70fc-42c7-8b72-3ca70d3f08b2","Type":"ContainerDied","Data":"0d95e0a4cc8afe8399be4f013bb428059c6955b0b0819cb8da662b296730c2fa"}
Oct 30 03:40:01.584927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.584905    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d95e0a4cc8afe8399be4f013bb428059c6955b0b0819cb8da662b296730c2fa"
Oct 30 03:40:01.585147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:01.585133    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"
Oct 30 03:40:06.203259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.203213    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"]
Oct 30 03:40:06.205365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.205330    4877 cpu_manager.go:414] "containerMap: removing container" podUID="21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" containerName="revoke-keys"
Oct 30 03:40:06.205575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.205559    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" containerName="revoke-keys"
Oct 30 03:40:06.205756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.205743    4877 cpu_manager.go:414] "containerMap: removing container" podUID="47b9418b-70fc-42c7-8b72-3ca70d3f08b2" containerName="revoke-keys"
Oct 30 03:40:06.205856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.205839    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b9418b-70fc-42c7-8b72-3ca70d3f08b2" containerName="revoke-keys"
Oct 30 03:40:06.206275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.206252    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b9418b-70fc-42c7-8b72-3ca70d3f08b2" containerName="revoke-keys"
Oct 30 03:40:06.206453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.206440    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" containerName="revoke-keys"
Oct 30 03:40:06.211968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.211940    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.226165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.226132    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"]
Oct 30 03:40:06.231811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.231780    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:40:06.231986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.231809    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:40:06.296323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.296276    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.296323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.296327    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvgr\" (UniqueName: \"kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.296567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.296533    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.397788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.397754    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.397788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.397791    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvgr\" (UniqueName: \"kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.398059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.397868    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.398537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.398518    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.400052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.400029    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.416081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.416052    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvgr\" (UniqueName: \"kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr\") pod \"maas-api-revoke-sub-45320976-5br2f\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.530046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.530009    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:06.843777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:06.843737    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"]
Oct 30 03:40:06.844033 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:40:06.844006    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5009972_a6ad_4988_909f_260e3c8b1c6c.slice/crio-b48a49ae9e3e6ff4737bafc3df9f2b2a3a1eadb4ed0596d44e424af96a3c33d6 WatchSource:0}: Error finding container b48a49ae9e3e6ff4737bafc3df9f2b2a3a1eadb4ed0596d44e424af96a3c33d6: Status 404 returned error can't find the container with id b48a49ae9e3e6ff4737bafc3df9f2b2a3a1eadb4ed0596d44e424af96a3c33d6
Oct 30 03:40:07.618103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.618066    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"]
Oct 30 03:40:07.622660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.622632    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.641132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.641105    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"]
Oct 30 03:40:07.642400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.642369    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f" event={"ID":"e5009972-a6ad-4988-909f-260e3c8b1c6c","Type":"ContainerStarted","Data":"1c87c31920a3975078e5d9864a43f73f98644dd13370b2096b5a8bd417356d40"}
Oct 30 03:40:07.642568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.642437    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f" event={"ID":"e5009972-a6ad-4988-909f-260e3c8b1c6c","Type":"ContainerStarted","Data":"b48a49ae9e3e6ff4737bafc3df9f2b2a3a1eadb4ed0596d44e424af96a3c33d6"}
Oct 30 03:40:07.698932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.698881    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f" podStartSLOduration=1.698866684 podStartE2EDuration="1.698866684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:07.694292053 +0000 UTC m=+2236.123181479" watchObservedRunningTime="2026-10-30 03:40:07.698866684 +0000 UTC m=+2236.127756103"
Oct 30 03:40:07.715852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.715816    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.715852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.715855    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.716115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.716044    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq825\" (UniqueName: \"kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.817124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.817083    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.817263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.817130    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.817263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.817235    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq825\" (UniqueName: \"kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.817894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.817830    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.819385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.819364    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.836192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.836158    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq825\" (UniqueName: \"kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825\") pod \"maas-api-revoke-sub-c58e48da-79r44\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:07.943000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:07.942903    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:08.268983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:08.266759    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"]
Oct 30 03:40:08.651137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:08.651099    4877 generic.go:356] "Generic (PLEG): container finished" podID="e5009972-a6ad-4988-909f-260e3c8b1c6c" containerID="1c87c31920a3975078e5d9864a43f73f98644dd13370b2096b5a8bd417356d40" exitCode=0
Oct 30 03:40:08.651882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:08.651177    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f" event={"ID":"e5009972-a6ad-4988-909f-260e3c8b1c6c","Type":"ContainerDied","Data":"1c87c31920a3975078e5d9864a43f73f98644dd13370b2096b5a8bd417356d40"}
Oct 30 03:40:08.653251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:08.653227    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44" event={"ID":"c0ccca04-f4e0-4c3d-aaa8-213934cec323","Type":"ContainerStarted","Data":"a84435a59716e8a0038389f64a09d2cb85c62ce41b298f306d06ea5bd683c858"}
Oct 30 03:40:08.653445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:08.653272    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44" event={"ID":"c0ccca04-f4e0-4c3d-aaa8-213934cec323","Type":"ContainerStarted","Data":"f55642509d491315753088228cecf28ff35f373662e6cd4e96d429e0b6a6c6b0"}
Oct 30 03:40:08.732582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:08.732526    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44" podStartSLOduration=1.732510085 podStartE2EDuration="1.732510085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:08.72954054 +0000 UTC m=+2237.158429958" watchObservedRunningTime="2026-10-30 03:40:08.732510085 +0000 UTC m=+2237.161399503"
Oct 30 03:40:09.672895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.672858    4877 generic.go:356] "Generic (PLEG): container finished" podID="c0ccca04-f4e0-4c3d-aaa8-213934cec323" containerID="a84435a59716e8a0038389f64a09d2cb85c62ce41b298f306d06ea5bd683c858" exitCode=0
Oct 30 03:40:09.673383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.673090    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44" event={"ID":"c0ccca04-f4e0-4c3d-aaa8-213934cec323","Type":"ContainerDied","Data":"a84435a59716e8a0038389f64a09d2cb85c62ce41b298f306d06ea5bd683c858"}
Oct 30 03:40:09.889329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.889298    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:09.945005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.944917    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca\") pod \"e5009972-a6ad-4988-909f-260e3c8b1c6c\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") "
Oct 30 03:40:09.945222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.945034    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr\" (UniqueName: \"kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr\") pod \"e5009972-a6ad-4988-909f-260e3c8b1c6c\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") "
Oct 30 03:40:09.945222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.945115    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca\") pod \"e5009972-a6ad-4988-909f-260e3c8b1c6c\" (UID: \"e5009972-a6ad-4988-909f-260e3c8b1c6c\") "
Oct 30 03:40:09.945773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.945741    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca" pod "e5009972-a6ad-4988-909f-260e3c8b1c6c" (UID: "e5009972-a6ad-4988-909f-260e3c8b1c6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:09.947757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.947728    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca" pod "e5009972-a6ad-4988-909f-260e3c8b1c6c" (UID: "e5009972-a6ad-4988-909f-260e3c8b1c6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:09.948398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:09.948372    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr" pod "e5009972-a6ad-4988-909f-260e3c8b1c6c" (UID: "e5009972-a6ad-4988-909f-260e3c8b1c6c"). InnerVolumeSpecName "kube-api-access-2cvgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:10.046859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.046821    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:10.046859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.046860    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvgr\" (UniqueName: \"kubernetes.io/projected/e5009972-a6ad-4988-909f-260e3c8b1c6c-kube-api-access-2cvgr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:10.047074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.046877    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5009972-a6ad-4988-909f-260e3c8b1c6c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:10.682230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.682199    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"
Oct 30 03:40:10.682683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.682236    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f" event={"ID":"e5009972-a6ad-4988-909f-260e3c8b1c6c","Type":"ContainerDied","Data":"b48a49ae9e3e6ff4737bafc3df9f2b2a3a1eadb4ed0596d44e424af96a3c33d6"}
Oct 30 03:40:10.682683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.682288    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b48a49ae9e3e6ff4737bafc3df9f2b2a3a1eadb4ed0596d44e424af96a3c33d6"
Oct 30 03:40:10.843416 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.843391    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:10.962761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.962684    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825\" (UniqueName: \"kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825\") pod \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") "
Oct 30 03:40:10.962761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.962722    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca\") pod \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") "
Oct 30 03:40:10.962944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.962922    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca\") pod \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\" (UID: \"c0ccca04-f4e0-4c3d-aaa8-213934cec323\") "
Oct 30 03:40:10.963252 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.963221    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca" pod "c0ccca04-f4e0-4c3d-aaa8-213934cec323" (UID: "c0ccca04-f4e0-4c3d-aaa8-213934cec323"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:10.963391 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.963325    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:10.964761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.964736    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825" pod "c0ccca04-f4e0-4c3d-aaa8-213934cec323" (UID: "c0ccca04-f4e0-4c3d-aaa8-213934cec323"). InnerVolumeSpecName "kube-api-access-jq825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:10.964846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:10.964791    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca" pod "c0ccca04-f4e0-4c3d-aaa8-213934cec323" (UID: "c0ccca04-f4e0-4c3d-aaa8-213934cec323"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:11.064974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:11.064932    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq825\" (UniqueName: \"kubernetes.io/projected/c0ccca04-f4e0-4c3d-aaa8-213934cec323-kube-api-access-jq825\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:11.064974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:11.064974    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0ccca04-f4e0-4c3d-aaa8-213934cec323-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:11.695891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:11.695858    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"
Oct 30 03:40:11.702258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:11.702228    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44" event={"ID":"c0ccca04-f4e0-4c3d-aaa8-213934cec323","Type":"ContainerDied","Data":"f55642509d491315753088228cecf28ff35f373662e6cd4e96d429e0b6a6c6b0"}
Oct 30 03:40:11.702463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:11.702273    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55642509d491315753088228cecf28ff35f373662e6cd4e96d429e0b6a6c6b0"
Oct 30 03:40:16.229250 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:40:16.229206    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:40:18.364027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.363986    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"]
Oct 30 03:40:18.364899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.364878    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c0ccca04-f4e0-4c3d-aaa8-213934cec323" containerName="revoke-keys"
Oct 30 03:40:18.365024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.364900    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ccca04-f4e0-4c3d-aaa8-213934cec323" containerName="revoke-keys"
Oct 30 03:40:18.365024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.364955    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e5009972-a6ad-4988-909f-260e3c8b1c6c" containerName="revoke-keys"
Oct 30 03:40:18.365024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.364964    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5009972-a6ad-4988-909f-260e3c8b1c6c" containerName="revoke-keys"
Oct 30 03:40:18.365190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.365141    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5009972-a6ad-4988-909f-260e3c8b1c6c" containerName="revoke-keys"
Oct 30 03:40:18.365190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.365167    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ccca04-f4e0-4c3d-aaa8-213934cec323" containerName="revoke-keys"
Oct 30 03:40:18.382448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.376678    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.426201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.426160    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"]
Oct 30 03:40:18.431850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.431714    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-35f43f-openshift-default-dockercfg-dt5f6\""
Oct 30 03:40:18.484478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484264    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484317    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484364    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484534    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484653    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484687    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484724    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484764    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8vk\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.484856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.484812    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589274    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589457    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589497    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589551    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589606    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8vk\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589692    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589844    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589871    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.589923    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.590594    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.590927    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.591238    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.594442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.592052    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.600342 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.600307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.601184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.601155    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.622509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.613993    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.629628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.629596    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.635809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.635778    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8vk\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk\") pod \"e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") " pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:18.748077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:18.748040    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:19.139082 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.139059    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"]
Oct 30 03:40:19.139267 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:40:19.139235    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8566642_4ab2_4ded_ae1a_75bc8d2fb796.slice/crio-cdd8fe4c2a01d0655b3d1eaf7d23cb0800674e786a9bfbbc2a5cae4d432702c6 WatchSource:0}: Error finding container cdd8fe4c2a01d0655b3d1eaf7d23cb0800674e786a9bfbbc2a5cae4d432702c6: Status 404 returned error can't find the container with id cdd8fe4c2a01d0655b3d1eaf7d23cb0800674e786a9bfbbc2a5cae4d432702c6
Oct 30 03:40:19.142655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.142620    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:19.142747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.142683    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:19.142747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.142712    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:19.763098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.763059    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" event={"ID":"b8566642-4ab2-4ded-ae1a-75bc8d2fb796","Type":"ContainerStarted","Data":"703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"}
Oct 30 03:40:19.763579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.763401    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" event={"ID":"b8566642-4ab2-4ded-ae1a-75bc8d2fb796","Type":"ContainerStarted","Data":"cdd8fe4c2a01d0655b3d1eaf7d23cb0800674e786a9bfbbc2a5cae4d432702c6"}
Oct 30 03:40:19.811446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:19.809856    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podStartSLOduration=1.809837141 podStartE2EDuration="1.809837141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:19.806127233 +0000 UTC m=+2248.235016681" watchObservedRunningTime="2026-10-30 03:40:19.809837141 +0000 UTC m=+2248.238726561"
Oct 30 03:40:20.748488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:20.748416    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:20.754790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:20.754758    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:20.771171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:20.771142    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:20.772171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:20.772149    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:40:25.038834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.038794    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"]
Oct 30 03:40:25.044653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.044634    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.078724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.078683    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"]
Oct 30 03:40:25.155869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.155827    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e10c0f-openshift-default-dockercfg-gj6dh\""
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238584    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238654    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238736    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238781    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcb8\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238887    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238932    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.238979    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.239014    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.253448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.239129    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.340861    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.340931    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341014    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341054    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcb8\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341145    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341188    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341232    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341265    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341361    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.341983    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.342287    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.342998    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.343324    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.343635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.343617    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.354135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.354072    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.354476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.354449    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.363983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.363949    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcb8\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.365633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.365588    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.409048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.409011    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:25.413581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.413539    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:40:25.413738 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.413605    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:40:25.747072 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:40:25.747037    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4348e6_070a_4199_952b_58649a0ef0f6.slice/crio-fb1d836e3e8b059329c191a295c7b0c8b7364f6e81bfe140f949d9b19c5eb4fc WatchSource:0}: Error finding container fb1d836e3e8b059329c191a295c7b0c8b7364f6e81bfe140f949d9b19c5eb4fc: Status 404 returned error can't find the container with id fb1d836e3e8b059329c191a295c7b0c8b7364f6e81bfe140f949d9b19c5eb4fc
Oct 30 03:40:25.747332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.747301    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"]
Oct 30 03:40:25.749768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.749733    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:25.749871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.749821    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:25.749871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.749852    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:25.842469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.842410    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" event={"ID":"bf4348e6-070a-4199-952b-58649a0ef0f6","Type":"ContainerStarted","Data":"d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"}
Oct 30 03:40:25.842663 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.842499    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" event={"ID":"bf4348e6-070a-4199-952b-58649a0ef0f6","Type":"ContainerStarted","Data":"fb1d836e3e8b059329c191a295c7b0c8b7364f6e81bfe140f949d9b19c5eb4fc"}
Oct 30 03:40:25.887449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:25.885142    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podStartSLOduration=0.885124578 podStartE2EDuration="885.124578ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:25.881269119 +0000 UTC m=+2254.310158536" watchObservedRunningTime="2026-10-30 03:40:25.885124578 +0000 UTC m=+2254.314013999"
Oct 30 03:40:26.409455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:26.409400    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:26.416914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:26.416886    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:26.853574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:26.853542    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:26.854660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:26.854639    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:40:28.918166 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:28.918134    4877 generic.go:356] "Generic (PLEG): container finished" podID="300ee1d9-af02-4b23-93e4-d90143518f26" containerID="ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6" exitCode=0
Oct 30 03:40:28.918683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:28.918216    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" event={"ID":"300ee1d9-af02-4b23-93e4-d90143518f26","Type":"ContainerDied","Data":"ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"}
Oct 30 03:40:28.918683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:28.918673    4877 scope.go:122] "RemoveContainer" containerID="ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"
Oct 30 03:40:28.920348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:28.920315    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:28.920483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:28.920389    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:28.920483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:28.920448    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:29.929131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:29.929093    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" event={"ID":"300ee1d9-af02-4b23-93e4-d90143518f26","Type":"ContainerStarted","Data":"42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c"}
Oct 30 03:40:30.450038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:30.450002    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:40:30.450351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:30.450327    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 03:40:30.450448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:30.450399    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 03:40:31.450934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:31.450899    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 03:40:31.451344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:31.450959    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 03:40:32.450942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:32.450907    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 03:40:32.451363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:32.450958    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 03:40:32.606599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:32.606559    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"]
Oct 30 03:40:32.606838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:32.606805    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy" containerID="cri-o://9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d" gracePeriod=30
Oct 30 03:40:33.450584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.450542    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 03:40:33.450767 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.450598    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 03:40:33.630168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.630118    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"]
Oct 30 03:40:33.636610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.636584    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.656260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.656229    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"]
Oct 30 03:40:33.662247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.662222    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-93b821-openshift-default-dockercfg-rjxg8\""
Oct 30 03:40:33.712254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.712157    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.712556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.712531    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.712749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.712733    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.712906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.712888    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.713055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.713034    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.713183 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.713167    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.713319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.713301    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.713513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.713495    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m97x\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.713664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.713646    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815335    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815439    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815502    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815616    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815655    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815681    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815762    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815862    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m97x\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.815906    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.816806    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.817111    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.817690    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.818446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.818098    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.838449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.831751    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.858832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.858417    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.862026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.861994    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m97x\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.875124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.875090    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.876056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.876028    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo\") pod \"e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") " pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:33.958275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:33.958242    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:34.362379 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:40:34.362325    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd818f979_ebd7_448b_b761_3aa7d712a85c.slice/crio-01161ccf85e2dbb43a80a383f2b95e4f486f8cd681d7d28851bad402118c09af WatchSource:0}: Error finding container 01161ccf85e2dbb43a80a383f2b95e4f486f8cd681d7d28851bad402118c09af: Status 404 returned error can't find the container with id 01161ccf85e2dbb43a80a383f2b95e4f486f8cd681d7d28851bad402118c09af
Oct 30 03:40:34.363707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:34.363676    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"]
Oct 30 03:40:34.365745 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:34.365705    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:34.365858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:34.365792    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:34.365858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:34.365837    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:34.450648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:34.450620    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 03:40:34.450782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:34.450667    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 03:40:35.040926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.040886    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" event={"ID":"d818f979-ebd7-448b-b761-3aa7d712a85c","Type":"ContainerStarted","Data":"a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"}
Oct 30 03:40:35.041369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.040961    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" event={"ID":"d818f979-ebd7-448b-b761-3aa7d712a85c","Type":"ContainerStarted","Data":"01161ccf85e2dbb43a80a383f2b95e4f486f8cd681d7d28851bad402118c09af"}
Oct 30 03:40:35.084277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.084224    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podStartSLOduration=2.084205881 podStartE2EDuration="2.084205881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:35.080188143 +0000 UTC m=+2263.509077584" watchObservedRunningTime="2026-10-30 03:40:35.084205881 +0000 UTC m=+2263.513095299"
Oct 30 03:40:35.455521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.455412    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:40:35.456004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.455936    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:40:35.456660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.456641    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 03:40:35.959346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.959307    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:35.965246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:35.965218    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:36.050410 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:36.050380    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:36.051397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:36.051371    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:40:36.125729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:36.125694    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"]
Oct 30 03:40:37.974212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:37.974180    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:40:38.019468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019396    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.019643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019509    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.019643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019582    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.019791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019653    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.019791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019698    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:38.019791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019710    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.019791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019766    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.020001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019944    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.020001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.019967    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:38.020130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020019    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.020130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020091    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token\") pod \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\" (UID: \"f28cb7f0-a115-442a-bdf3-d4969c0219b1\") "
Oct 30 03:40:38.020234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020205    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:38.020295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020272    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:38.020477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020452    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:38.020831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020775    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.020831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020800    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.020831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020814    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.020831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020827    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.021117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.020839    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.022236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.022195    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:40:38.022326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.022298    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:38.022752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.022709    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "kube-api-access-5zjjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:38.022890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.022869    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token" pod "f28cb7f0-a115-442a-bdf3-d4969c0219b1" (UID: "f28cb7f0-a115-442a-bdf3-d4969c0219b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:38.068868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.068828    4877 generic.go:356] "Generic (PLEG): container finished" podID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerID="9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d" exitCode=0
Oct 30 03:40:38.069060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.068931    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"
Oct 30 03:40:38.069060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.068969    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" event={"ID":"f28cb7f0-a115-442a-bdf3-d4969c0219b1","Type":"ContainerDied","Data":"9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d"}
Oct 30 03:40:38.069060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.069035    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4" event={"ID":"f28cb7f0-a115-442a-bdf3-d4969c0219b1","Type":"ContainerDied","Data":"6633b76735789baac623370db69153b1025e8756621fb1ae9ddfeba2a19348d8"}
Oct 30 03:40:38.069214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.069067    4877 scope.go:122] "RemoveContainer" containerID="9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d"
Oct 30 03:40:38.082049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.082017    4877 scope.go:122] "RemoveContainer" containerID="9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d"
Oct 30 03:40:38.082381 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:40:38.082356    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d\": container with ID starting with 9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d not found: ID does not exist" containerID="9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d"
Oct 30 03:40:38.082523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.082390    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d"} err="failed to get container status \"9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d\": rpc error: code = NotFound desc = could not find container \"9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d\": container with ID starting with 9e11811a2335b3897e682f4ad9cb59dbf8844f03956e659825d2a296e41f851d not found: ID does not exist"
Oct 30 03:40:38.121403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.121372    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"]
Oct 30 03:40:38.122284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.122257    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.122284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.122286    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.122464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.122300    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28cb7f0-a115-442a-bdf3-d4969c0219b1-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.122464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.122316    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjjm\" (UniqueName: \"kubernetes.io/projected/f28cb7f0-a115-442a-bdf3-d4969c0219b1-kube-api-access-5zjjm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:38.125602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:38.125579    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-f4vw4"]
Oct 30 03:40:39.701861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:39.701827    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" path="/var/lib/kubelet/pods/f28cb7f0-a115-442a-bdf3-d4969c0219b1/volumes"
Oct 30 03:40:46.232698 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:40:46.232658    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:40:46.628241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.628207    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"]
Oct 30 03:40:46.629219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.629196    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy"
Oct 30 03:40:46.629328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.629220    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy"
Oct 30 03:40:46.629454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.629411    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28cb7f0-a115-442a-bdf3-d4969c0219b1" containerName="istio-proxy"
Oct 30 03:40:46.634351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.634323    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.648591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.648559    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-398c25-openshift-default-dockercfg-88fjr\""
Oct 30 03:40:46.667268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.667134    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"]
Oct 30 03:40:46.712442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712394    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712527    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712588    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712609    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712733    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712750    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712874    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.712937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.712914    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thd6\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.713052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.713039    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839414    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839623    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839689    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839717    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839776    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839800    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.839977    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.840019    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thd6\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.840864    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.841124    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.842028    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.843442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.842997    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.847002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.844624    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.848369 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.848345    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.851207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.851182    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.880254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.880106    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.881399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.881331    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thd6\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:46.956176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:46.956138    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:47.286887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:47.286856    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"]
Oct 30 03:40:47.289224 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:40:47.289179    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828f9787_d0fc_4aad_8c2d_04f2a21cf3a9.slice/crio-5a495c81058ea786fac1b9af40f01d54ff13e4be089bb4ae83f7b7d97019cf28 WatchSource:0}: Error finding container 5a495c81058ea786fac1b9af40f01d54ff13e4be089bb4ae83f7b7d97019cf28: Status 404 returned error can't find the container with id 5a495c81058ea786fac1b9af40f01d54ff13e4be089bb4ae83f7b7d97019cf28
Oct 30 03:40:47.292381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:47.292338    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:47.292513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:47.292472    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:47.292579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:47.292518    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:40:48.295443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:48.295185    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" event={"ID":"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9","Type":"ContainerStarted","Data":"8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"}
Oct 30 03:40:48.295443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:48.295274    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" event={"ID":"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9","Type":"ContainerStarted","Data":"5a495c81058ea786fac1b9af40f01d54ff13e4be089bb4ae83f7b7d97019cf28"}
Oct 30 03:40:48.957031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:48.957004    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:48.962793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:48.962763    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:49.005986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:49.005931    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podStartSLOduration=3.005916882 podStartE2EDuration="3.005916882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:48.361843113 +0000 UTC m=+2276.790732547" watchObservedRunningTime="2026-10-30 03:40:49.005916882 +0000 UTC m=+2277.434806299"
Oct 30 03:40:49.303651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:49.303613    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:49.304954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:49.304918    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:40:51.158662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.158623    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298" podUID="d59a9592-12c4-465a-bc9c-81ca22ff3118" containerName="main" containerID="cri-o://f0aeb1fe1c252ec6ce67e1a744606ecf4eb1852859090ab7897ed08859ea76e8" gracePeriod=45
Oct 30 03:40:51.325182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.325139    4877 generic.go:356] "Generic (PLEG): container finished" podID="d59a9592-12c4-465a-bc9c-81ca22ff3118" containerID="f0aeb1fe1c252ec6ce67e1a744606ecf4eb1852859090ab7897ed08859ea76e8" exitCode=0
Oct 30 03:40:51.325570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.325501    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298" event={"ID":"d59a9592-12c4-465a-bc9c-81ca22ff3118","Type":"ContainerDied","Data":"f0aeb1fe1c252ec6ce67e1a744606ecf4eb1852859090ab7897ed08859ea76e8"}
Oct 30 03:40:51.456102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.456072    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:40:51.634298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634261    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache\") pod \"d59a9592-12c4-465a-bc9c-81ca22ff3118\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") "
Oct 30 03:40:51.634526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634381    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs\" (UniqueName: \"kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs\") pod \"d59a9592-12c4-465a-bc9c-81ca22ff3118\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") "
Oct 30 03:40:51.634604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634566    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486\" (UniqueName: \"kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486\") pod \"d59a9592-12c4-465a-bc9c-81ca22ff3118\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") "
Oct 30 03:40:51.634604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634575    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache" pod "d59a9592-12c4-465a-bc9c-81ca22ff3118" (UID: "d59a9592-12c4-465a-bc9c-81ca22ff3118"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:51.634723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634631    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm\") pod \"d59a9592-12c4-465a-bc9c-81ca22ff3118\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") "
Oct 30 03:40:51.634723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634701    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir\") pod \"d59a9592-12c4-465a-bc9c-81ca22ff3118\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") "
Oct 30 03:40:51.634836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634727    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home\") pod \"d59a9592-12c4-465a-bc9c-81ca22ff3118\" (UID: \"d59a9592-12c4-465a-bc9c-81ca22ff3118\") "
Oct 30 03:40:51.634946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.634919    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir" pod "d59a9592-12c4-465a-bc9c-81ca22ff3118" (UID: "d59a9592-12c4-465a-bc9c-81ca22ff3118"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:51.635053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.635032    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home" pod "d59a9592-12c4-465a-bc9c-81ca22ff3118" (UID: "d59a9592-12c4-465a-bc9c-81ca22ff3118"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:51.635535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.635512    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:51.635601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.635540    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:51.635601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.635553    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:51.636793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.636763    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs" pod "d59a9592-12c4-465a-bc9c-81ca22ff3118" (UID: "d59a9592-12c4-465a-bc9c-81ca22ff3118"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:51.636884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.636799    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm" pod "d59a9592-12c4-465a-bc9c-81ca22ff3118" (UID: "d59a9592-12c4-465a-bc9c-81ca22ff3118"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:51.636884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.636810    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486" pod "d59a9592-12c4-465a-bc9c-81ca22ff3118" (UID: "d59a9592-12c4-465a-bc9c-81ca22ff3118"). InnerVolumeSpecName "kube-api-access-lz486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:51.737504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.737467    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59a9592-12c4-465a-bc9c-81ca22ff3118-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:51.737504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.737494    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz486\" (UniqueName: \"kubernetes.io/projected/d59a9592-12c4-465a-bc9c-81ca22ff3118-kube-api-access-lz486\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:51.737504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:51.737503    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59a9592-12c4-465a-bc9c-81ca22ff3118-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:40:52.336584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:52.336550    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298" event={"ID":"d59a9592-12c4-465a-bc9c-81ca22ff3118","Type":"ContainerDied","Data":"81e5d0de994001ded29de1c33c399bd2f762f09a67ab1d116990cf4a62b9bfa3"}
Oct 30 03:40:52.336584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:52.336568    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"
Oct 30 03:40:52.337113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:52.336607    4877 scope.go:122] "RemoveContainer" containerID="f0aeb1fe1c252ec6ce67e1a744606ecf4eb1852859090ab7897ed08859ea76e8"
Oct 30 03:40:52.393815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:52.393782    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"]
Oct 30 03:40:52.403834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:52.403804    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-817531/auth-test-model-817531-kserve-864c5567bf-s2298"]
Oct 30 03:40:53.699923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:53.699890    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59a9592-12c4-465a-bc9c-81ca22ff3118" path="/var/lib/kubelet/pods/d59a9592-12c4-465a-bc9c-81ca22ff3118/volumes"
Oct 30 03:40:55.043002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:40:55.042957    4877 scope.go:122] "RemoveContainer" containerID="6f7d75549689c9bcb4b030ee392ba396f27c8e56a93f8944a769c27b2c0bfa84"
Oct 30 03:41:16.307533 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:41:16.307480    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:41:20.768921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.768835    4877 generic.go:356] "Generic (PLEG): container finished" podID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerID="703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286" exitCode=0
Oct 30 03:41:20.768921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.768907    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" event={"ID":"b8566642-4ab2-4ded-ae1a-75bc8d2fb796","Type":"ContainerDied","Data":"703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"}
Oct 30 03:41:20.769504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.769378    4877 scope.go:122] "RemoveContainer" containerID="703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"
Oct 30 03:41:20.771036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771019    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:41:20.771406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771380    4877 generic.go:356] "Generic (PLEG): container finished" podID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerID="d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c" exitCode=0
Oct 30 03:41:20.771567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771452    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" event={"ID":"bf4348e6-070a-4199-952b-58649a0ef0f6","Type":"ContainerDied","Data":"d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"}
Oct 30 03:41:20.771567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771529    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:20.771713 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771599    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:20.771713 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771634    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:20.771852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.771788    4877 scope.go:122] "RemoveContainer" containerID="d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"
Oct 30 03:41:20.773533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.773504    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:20.773628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.773574    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:20.773628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:20.773613    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:21.836448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:21.833383    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" event={"ID":"bf4348e6-070a-4199-952b-58649a0ef0f6","Type":"ContainerStarted","Data":"58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693"}
Oct 30 03:41:21.838471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:21.838440    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" event={"ID":"b8566642-4ab2-4ded-ae1a-75bc8d2fb796","Type":"ContainerStarted","Data":"9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f"}
Oct 30 03:41:22.412626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:22.412595    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:41:22.433477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:22.431003    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 03:41:22.433477 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:22.431067    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 03:41:22.749497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:22.748991    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:41:22.753599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:22.753566    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 03:41:22.753764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:22.753625    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 03:41:23.422447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:23.409567    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 03:41:23.422447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:23.409626    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 03:41:23.749197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:23.749114    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 03:41:23.749197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:23.749165    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 03:41:24.410871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:24.410768    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 03:41:24.411176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:24.410841    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 03:41:24.748837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:24.748748    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 03:41:24.748837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:24.748809    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 03:41:25.410452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:25.409761    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 03:41:25.410452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:25.409825    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 03:41:25.748485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:25.748380    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 03:41:25.748485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:25.748470    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 03:41:26.411457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:26.410706    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 03:41:26.411457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:26.410783    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 03:41:26.748555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:26.748453    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 03:41:26.748555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:26.748518    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 03:41:26.854204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:26.854167    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:41:27.474167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.474127    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:41:27.477226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.477198    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 03:41:27.745047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.744010    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"]
Oct 30 03:41:27.745624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.745365    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d59a9592-12c4-465a-bc9c-81ca22ff3118" containerName="main"
Oct 30 03:41:27.745624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.745392    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59a9592-12c4-465a-bc9c-81ca22ff3118" containerName="main"
Oct 30 03:41:27.745801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.745648    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59a9592-12c4-465a-bc9c-81ca22ff3118" containerName="main"
Oct 30 03:41:27.749826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.749806    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.753102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.753072    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:41:27.753353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.753336    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:41:27.753959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.753940    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 03:41:27.764163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.764140    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"payload-processing-external-model-e2e--154314f1-dockercfg-k6mhx\""
Oct 30 03:41:27.764271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.764154    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"agc-cred-354f9bd61522\""
Oct 30 03:41:27.764361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.764339    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-35f43f\""
Oct 30 03:41:27.764505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.764395    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"kube-root-ca.crt\""
Oct 30 03:41:27.764505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.764433    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"routing-overlay\""
Oct 30 03:41:27.764593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.764539    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"openshift-service-ca.crt\""
Oct 30 03:41:27.765878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.765856    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"]
Oct 30 03:41:27.823010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.822964    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.823501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.823469    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.823671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.823570    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4495w\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.823671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.823640    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.924585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.924548    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4495w\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.924769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.924615    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.924769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.924659    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.924861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.924818    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.925491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.925459    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.925723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.925698    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.927207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.927189    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:27.943418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:27.943392    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4495w\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:28.067198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:28.065699    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:28.396697 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:41:28.396664    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5a11a1_e0b6_41a0_997a_b76c9e968674.slice/crio-3da89ac8dd7babab5070d530304d336f66b42167d5c17f4bc291ec2bba3f95b9 WatchSource:0}: Error finding container 3da89ac8dd7babab5070d530304d336f66b42167d5c17f4bc291ec2bba3f95b9: Status 404 returned error can't find the container with id 3da89ac8dd7babab5070d530304d336f66b42167d5c17f4bc291ec2bba3f95b9
Oct 30 03:41:28.398585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:28.398558    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"]
Oct 30 03:41:28.917591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:28.917552    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4" event={"ID":"2b5a11a1-e0b6-41a0-997a-b76c9e968674","Type":"ContainerStarted","Data":"22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488"}
Oct 30 03:41:28.918069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:28.917610    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4" event={"ID":"2b5a11a1-e0b6-41a0-997a-b76c9e968674","Type":"ContainerStarted","Data":"3da89ac8dd7babab5070d530304d336f66b42167d5c17f4bc291ec2bba3f95b9"}
Oct 30 03:41:28.918069 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:28.917919    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:28.963588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:28.960123    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4" podStartSLOduration=1.9601050930000001 podStartE2EDuration="1.960105093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:28.954745109 +0000 UTC m=+2317.383634528" watchObservedRunningTime="2026-10-30 03:41:28.960105093 +0000 UTC m=+2317.388994530"
Oct 30 03:41:31.173771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.173740    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"]
Oct 30 03:41:31.178731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.178707    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.222894 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.222859    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"]
Oct 30 03:41:31.226764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.226738    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-372480-openshift-default-dockercfg-564ss\""
Oct 30 03:41:31.290208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290171    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqgb\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290217    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290324    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290389    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290479    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290642 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290513    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290675    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290697    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.290821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.290789    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.400883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.400842    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.401185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.401164    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.401302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.401286    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.401405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.401392    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.403609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.403582    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.404534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.404466    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.408847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.408815    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.413608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqgb\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.413653    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.402171    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.408717    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.402396    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.404395    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.417438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.401932    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.436488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.430035    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.436488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.430307    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.466450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.445346    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.467935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.467897    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqgb\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.536578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.523034    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:31.854110 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:41:31.854080    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3bd079_fc0a_4671_b826_b2369583768d.slice/crio-213c7933e2e81cac05135e9ae79c161a78ca234cd9b460a99490c63f2e528e09 WatchSource:0}: Error finding container 213c7933e2e81cac05135e9ae79c161a78ca234cd9b460a99490c63f2e528e09: Status 404 returned error can't find the container with id 213c7933e2e81cac05135e9ae79c161a78ca234cd9b460a99490c63f2e528e09
Oct 30 03:41:31.856878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.856855    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"]
Oct 30 03:41:31.856980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.856849    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:31.857023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.856973    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:31.857023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.857012    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:31.945292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:31.945260    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" event={"ID":"3d3bd079-fc0a-4671-b826-b2369583768d","Type":"ContainerStarted","Data":"213c7933e2e81cac05135e9ae79c161a78ca234cd9b460a99490c63f2e528e09"}
Oct 30 03:41:32.955052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:32.955016    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" event={"ID":"3d3bd079-fc0a-4671-b826-b2369583768d","Type":"ContainerStarted","Data":"8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"}
Oct 30 03:41:33.021509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:33.021444    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podStartSLOduration=2.021416264 podStartE2EDuration="2.021416264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:33.007264437 +0000 UTC m=+2321.436153855" watchObservedRunningTime="2026-10-30 03:41:33.021416264 +0000 UTC m=+2321.450305682"
Oct 30 03:41:33.523537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:33.523489    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:33.530540 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:33.530514    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:33.979006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:33.978928    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:33.988666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:33.988638    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:41:39.940558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:39.940356    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:41:46.232697 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:41:46.232657    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:41:47.232811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.232778    4877 generic.go:356] "Generic (PLEG): container finished" podID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerID="8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6" exitCode=0
Oct 30 03:41:47.233303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.232845    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" event={"ID":"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9","Type":"ContainerDied","Data":"8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"}
Oct 30 03:41:47.233377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.233325    4877 scope.go:122] "RemoveContainer" containerID="8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"
Oct 30 03:41:47.234848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.234828    4877 generic.go:356] "Generic (PLEG): container finished" podID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerID="a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5" exitCode=0
Oct 30 03:41:47.234940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.234900    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" event={"ID":"d818f979-ebd7-448b-b761-3aa7d712a85c","Type":"ContainerDied","Data":"a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"}
Oct 30 03:41:47.235245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.235219    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:47.235287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.235256    4877 scope.go:122] "RemoveContainer" containerID="a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"
Oct 30 03:41:47.235322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.235304    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:47.235357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.235339    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:47.236735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.236711    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:47.236795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.236773    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:47.236832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:47.236805    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:48.257377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.257335    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" event={"ID":"d818f979-ebd7-448b-b761-3aa7d712a85c","Type":"ContainerStarted","Data":"ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951"}
Oct 30 03:41:48.259787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.259758    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" event={"ID":"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9","Type":"ContainerStarted","Data":"a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d"}
Oct 30 03:41:48.957543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.957192    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:41:48.957543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.957241    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:41:48.957543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.957473    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:41:48.961902 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.961877    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:41:48.962728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.962692    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:41:48.962916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:48.962894    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:41:49.315597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:49.315562    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:41:49.963445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:49.957555    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:41:49.963445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:49.957618    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:41:49.963445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:49.958834    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:41:49.963445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:49.958878    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:41:50.957111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:50.957081    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:41:50.957530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:50.957497    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:41:50.959344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:50.959321    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:41:50.959462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:50.959362    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:41:51.050476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:51.050434    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:41:51.956823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:51.956786    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:41:51.956993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:51.956848    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:41:51.959093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:51.959069    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:41:51.959444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:51.959107    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:41:52.957739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:52.957705    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 03:41:52.957962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:52.957762    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 03:41:52.958977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:52.958951    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 03:41:52.959153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:52.958994    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 03:41:53.962052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:53.962020    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:41:53.962992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:53.962971    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 03:41:54.078100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:54.078056    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:41:54.080229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:54.080201    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 03:41:58.832410 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.832377    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"]
Oct 30 03:41:58.838197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.838166    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.852938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.852902    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-855e14-openshift-default-dockercfg-24jzw\""
Oct 30 03:41:58.873186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.873142    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"]
Oct 30 03:41:58.922675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.922636    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.922853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.922825    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.922919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.922863    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.922919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.922888    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.923043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.922948    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.923043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.923028    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqn89\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.923151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.923128    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.923210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.923186    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:58.923266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:58.923242    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.025452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.025390    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.050446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.044516    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.050446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.044575    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.050446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.044604    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.050446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.049820    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.050446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.026590    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.053459    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.053530    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqn89\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.053642    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.053712    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.053781    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.067837    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.073297    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.095999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.095810    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.108448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.105144    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.108448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.105400    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.113103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.113072    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.166204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.166163    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqn89\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.168931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.168690    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:41:59.534866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.534829    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"]
Oct 30 03:41:59.537044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.537006    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:59.537144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.537082    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:41:59.537185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:41:59.537145    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:42:00.440090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:00.439985    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" event={"ID":"ed33a285-7366-4359-93ff-1d4118773ab3","Type":"ContainerStarted","Data":"1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"}
Oct 30 03:42:00.440090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:00.440084    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" event={"ID":"ed33a285-7366-4359-93ff-1d4118773ab3","Type":"ContainerStarted","Data":"dacce38b1e1e5cf644ac773426c9453835913c77ba311986ce5804cbde88bae5"}
Oct 30 03:42:00.505638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:00.505583    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podStartSLOduration=2.505565619 podStartE2EDuration="2.505565619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:00.492275618 +0000 UTC m=+2348.921165109" watchObservedRunningTime="2026-10-30 03:42:00.505565619 +0000 UTC m=+2348.934455110"
Oct 30 03:42:01.169964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:01.169923    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:42:01.177173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:01.177139    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:42:01.451845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:01.451752    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:42:01.452812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:01.452787    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:42:16.232879 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:42:16.232839    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:42:32.808917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:32.808875    4877 generic.go:356] "Generic (PLEG): container finished" podID="3d3bd079-fc0a-4671-b826-b2369583768d" containerID="8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20" exitCode=0
Oct 30 03:42:32.809403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:32.808965    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" event={"ID":"3d3bd079-fc0a-4671-b826-b2369583768d","Type":"ContainerDied","Data":"8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"}
Oct 30 03:42:32.809556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:32.809538    4877 scope.go:122] "RemoveContainer" containerID="8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"
Oct 30 03:42:32.811566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:32.811534    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:42:32.811673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:32.811606    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:42:32.811673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:32.811651    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:42:33.824048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:33.823832    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" event={"ID":"3d3bd079-fc0a-4671-b826-b2369583768d","Type":"ContainerStarted","Data":"0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef"}
Oct 30 03:42:33.982441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:33.982212    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:42:34.523657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:34.523612    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:42:34.525294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:34.525266    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 03:42:34.525482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:34.525320    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 03:42:35.524499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:35.524461    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 03:42:35.524903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:35.524516    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 03:42:36.524597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:36.524562    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 03:42:36.525008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:36.524621    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 03:42:37.524100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:37.524059    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 03:42:37.524284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:37.524124    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 03:42:38.523959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:38.523925    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 03:42:38.524346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:38.523990    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 03:42:39.530290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:39.530259    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:42:39.531414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:39.531395    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 03:42:43.901847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.901802    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"]
Oct 30 03:42:43.907062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.907033    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:43.927648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.927619    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"]
Oct 30 03:42:43.931334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.931292    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:42:43.931727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.931704    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:42:43.948837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.948800    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2b4q\" (UniqueName: \"kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:43.949227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.949200    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:43.949600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:43.949563    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.051017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.050983    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2b4q\" (UniqueName: \"kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.051281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.051041    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.051281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.051136    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.051796 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.051770    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.053626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.053602    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.069654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.069630    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2b4q\" (UniqueName: \"kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q\") pod \"maas-api-revoke-sub-56426188-wrtz7\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.225809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.225726    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:44.532992 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:42:44.532960    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716aac94_a59f_44fb_a2f2_c4af7ac1680d.slice/crio-160573997fd20e87ed519907edadd62ceb00b108ec5ce6ceb556b6dc3f53dfed WatchSource:0}: Error finding container 160573997fd20e87ed519907edadd62ceb00b108ec5ce6ceb556b6dc3f53dfed: Status 404 returned error can't find the container with id 160573997fd20e87ed519907edadd62ceb00b108ec5ce6ceb556b6dc3f53dfed
Oct 30 03:42:44.537091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.537057    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"]
Oct 30 03:42:44.988251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.988203    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7" event={"ID":"716aac94-a59f-44fb-a2f2-c4af7ac1680d","Type":"ContainerStarted","Data":"9458a057611fbb69607c645cd1693dadbf54f40c68389a40655b09389bf32c30"}
Oct 30 03:42:44.988800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:44.988295    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7" event={"ID":"716aac94-a59f-44fb-a2f2-c4af7ac1680d","Type":"ContainerStarted","Data":"160573997fd20e87ed519907edadd62ceb00b108ec5ce6ceb556b6dc3f53dfed"}
Oct 30 03:42:45.030733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.030679    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7" podStartSLOduration=2.030664399 podStartE2EDuration="2.030664399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:45.028965244 +0000 UTC m=+2393.457854687" watchObservedRunningTime="2026-10-30 03:42:45.030664399 +0000 UTC m=+2393.459553816"
Oct 30 03:42:45.603343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.603300    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"]
Oct 30 03:42:45.609126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.609098    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.625468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.625412    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"]
Oct 30 03:42:45.675725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.675685    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8l8g\" (UniqueName: \"kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.675913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.675753    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.675977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.675950    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.777505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.777473    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8l8g\" (UniqueName: \"kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.777695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.777515    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.777695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.777593    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.778400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.778374    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.780108 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.780086    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.796638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.796606    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8l8g\" (UniqueName: \"kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g\") pod \"maas-api-revoke-sub-cb61edbd-4598x\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:45.925749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:45.925715    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:46.008480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:46.006593    4877 generic.go:356] "Generic (PLEG): container finished" podID="716aac94-a59f-44fb-a2f2-c4af7ac1680d" containerID="9458a057611fbb69607c645cd1693dadbf54f40c68389a40655b09389bf32c30" exitCode=0
Oct 30 03:42:46.008480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:46.006645    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7" event={"ID":"716aac94-a59f-44fb-a2f2-c4af7ac1680d","Type":"ContainerDied","Data":"9458a057611fbb69607c645cd1693dadbf54f40c68389a40655b09389bf32c30"}
Oct 30 03:42:46.230845 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:42:46.230745    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:42:46.237536 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:42:46.237504    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f272c3a_3525_407a_9f51_90cd882b79b9.slice/crio-2f78a44afa7766a5478f46d3aff834caeed04a412a3daed66a91bf87233b4407 WatchSource:0}: Error finding container 2f78a44afa7766a5478f46d3aff834caeed04a412a3daed66a91bf87233b4407: Status 404 returned error can't find the container with id 2f78a44afa7766a5478f46d3aff834caeed04a412a3daed66a91bf87233b4407
Oct 30 03:42:46.248090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:46.248055    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"]
Oct 30 03:42:47.020695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.020648    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x" event={"ID":"2f272c3a-3525-407a-9f51-90cd882b79b9","Type":"ContainerStarted","Data":"f9c1efbb8534661d662ccbf8fcbacd261a08de30d8d2ac6db82a47f59ba7470d"}
Oct 30 03:42:47.021212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.020722    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x" event={"ID":"2f272c3a-3525-407a-9f51-90cd882b79b9","Type":"ContainerStarted","Data":"2f78a44afa7766a5478f46d3aff834caeed04a412a3daed66a91bf87233b4407"}
Oct 30 03:42:47.067446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.067210    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x" podStartSLOduration=2.067194073 podStartE2EDuration="2.067194073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:47.06275404 +0000 UTC m=+2395.491643486" watchObservedRunningTime="2026-10-30 03:42:47.067194073 +0000 UTC m=+2395.496083491"
Oct 30 03:42:47.279648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.279590    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:47.405482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.405414    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca\") pod \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") "
Oct 30 03:42:47.405727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.405708    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q\" (UniqueName: \"kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q\") pod \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") "
Oct 30 03:42:47.405836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.405821    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca\") pod \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\" (UID: \"716aac94-a59f-44fb-a2f2-c4af7ac1680d\") "
Oct 30 03:42:47.406217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.406185    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca" pod "716aac94-a59f-44fb-a2f2-c4af7ac1680d" (UID: "716aac94-a59f-44fb-a2f2-c4af7ac1680d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:47.406375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.406358    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:42:47.408222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.408184    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca" pod "716aac94-a59f-44fb-a2f2-c4af7ac1680d" (UID: "716aac94-a59f-44fb-a2f2-c4af7ac1680d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:47.408406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.408387    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q" pod "716aac94-a59f-44fb-a2f2-c4af7ac1680d" (UID: "716aac94-a59f-44fb-a2f2-c4af7ac1680d"). InnerVolumeSpecName "kube-api-access-w2b4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:47.507886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.507850    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716aac94-a59f-44fb-a2f2-c4af7ac1680d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:42:47.507886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:47.507888    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2b4q\" (UniqueName: \"kubernetes.io/projected/716aac94-a59f-44fb-a2f2-c4af7ac1680d-kube-api-access-w2b4q\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:42:48.032051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:48.032014    4877 generic.go:356] "Generic (PLEG): container finished" podID="2f272c3a-3525-407a-9f51-90cd882b79b9" containerID="f9c1efbb8534661d662ccbf8fcbacd261a08de30d8d2ac6db82a47f59ba7470d" exitCode=0
Oct 30 03:42:48.032546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:48.032102    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x" event={"ID":"2f272c3a-3525-407a-9f51-90cd882b79b9","Type":"ContainerDied","Data":"f9c1efbb8534661d662ccbf8fcbacd261a08de30d8d2ac6db82a47f59ba7470d"}
Oct 30 03:42:48.033716 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:48.033699    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"
Oct 30 03:42:48.033832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:48.033741    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7" event={"ID":"716aac94-a59f-44fb-a2f2-c4af7ac1680d","Type":"ContainerDied","Data":"160573997fd20e87ed519907edadd62ceb00b108ec5ce6ceb556b6dc3f53dfed"}
Oct 30 03:42:48.033832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:48.033781    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="160573997fd20e87ed519907edadd62ceb00b108ec5ce6ceb556b6dc3f53dfed"
Oct 30 03:42:49.225363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.225338    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:49.332909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.332813    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca\") pod \"2f272c3a-3525-407a-9f51-90cd882b79b9\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") "
Oct 30 03:42:49.332909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.332875    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g\" (UniqueName: \"kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g\") pod \"2f272c3a-3525-407a-9f51-90cd882b79b9\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") "
Oct 30 03:42:49.333178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.332984    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca\") pod \"2f272c3a-3525-407a-9f51-90cd882b79b9\" (UID: \"2f272c3a-3525-407a-9f51-90cd882b79b9\") "
Oct 30 03:42:49.333294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.333262    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca" pod "2f272c3a-3525-407a-9f51-90cd882b79b9" (UID: "2f272c3a-3525-407a-9f51-90cd882b79b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:49.333740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.333715    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:42:49.335440 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.335398    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g" pod "2f272c3a-3525-407a-9f51-90cd882b79b9" (UID: "2f272c3a-3525-407a-9f51-90cd882b79b9"). InnerVolumeSpecName "kube-api-access-s8l8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:49.335538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.335510    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca" pod "2f272c3a-3525-407a-9f51-90cd882b79b9" (UID: "2f272c3a-3525-407a-9f51-90cd882b79b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:49.435321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.435287    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8l8g\" (UniqueName: \"kubernetes.io/projected/2f272c3a-3525-407a-9f51-90cd882b79b9-kube-api-access-s8l8g\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:42:49.435321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:49.435319    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f272c3a-3525-407a-9f51-90cd882b79b9-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:42:50.085235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:50.085175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x" event={"ID":"2f272c3a-3525-407a-9f51-90cd882b79b9","Type":"ContainerDied","Data":"2f78a44afa7766a5478f46d3aff834caeed04a412a3daed66a91bf87233b4407"}
Oct 30 03:42:50.085435 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:50.085293    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f78a44afa7766a5478f46d3aff834caeed04a412a3daed66a91bf87233b4407"
Oct 30 03:42:50.085435 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:50.085386    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"
Oct 30 03:42:51.152449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.152384    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"]
Oct 30 03:42:51.153089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.153073    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2f272c3a-3525-407a-9f51-90cd882b79b9" containerName="revoke-keys"
Oct 30 03:42:51.153089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.153089    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f272c3a-3525-407a-9f51-90cd882b79b9" containerName="revoke-keys"
Oct 30 03:42:51.153191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.153133    4877 cpu_manager.go:414] "containerMap: removing container" podUID="716aac94-a59f-44fb-a2f2-c4af7ac1680d" containerName="revoke-keys"
Oct 30 03:42:51.153191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.153139    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716aac94-a59f-44fb-a2f2-c4af7ac1680d" containerName="revoke-keys"
Oct 30 03:42:51.153257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.153241    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="716aac94-a59f-44fb-a2f2-c4af7ac1680d" containerName="revoke-keys"
Oct 30 03:42:51.153257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.153253    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f272c3a-3525-407a-9f51-90cd882b79b9" containerName="revoke-keys"
Oct 30 03:42:51.157101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.157081    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.178478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.178449    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"]
Oct 30 03:42:51.183338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.183305    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\""
Oct 30 03:42:51.183338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.183315    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-7e237115c4ea\""
Oct 30 03:42:51.183562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.183381    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-t5k2w\""
Oct 30 03:42:51.185845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.185819    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\""
Oct 30 03:42:51.185845 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.185832    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\""
Oct 30 03:42:51.186023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.185960    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\""
Oct 30 03:42:51.290457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.290380    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.290676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.290473    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.290676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.290518    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56t7\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.290861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.290733    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.392445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.392072    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.392445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.392152    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56t7\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.392445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.392436    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.392758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.392607    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.396444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.393844    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.396444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.394612    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.424492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.421693    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.432587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.427733    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56t7\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7\") pod \"payload-processing-external-model-e2e-extmod-b-372480-558769hv4\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.471334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.471301    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:51.801199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:51.801161    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"]
Oct 30 03:42:51.804473 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:42:51.804061    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889e3849_cc76_4b08_9a6c_d038b4e0fe5b.slice/crio-2276d1ee7b4ebc0ee5560e2c737f3b7e83e52366ff115224fb2301fac3c9b735 WatchSource:0}: Error finding container 2276d1ee7b4ebc0ee5560e2c737f3b7e83e52366ff115224fb2301fac3c9b735: Status 404 returned error can't find the container with id 2276d1ee7b4ebc0ee5560e2c737f3b7e83e52366ff115224fb2301fac3c9b735
Oct 30 03:42:52.102794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:52.102713    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4" event={"ID":"889e3849-cc76-4b08-9a6c-d038b4e0fe5b","Type":"ContainerStarted","Data":"a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629"}
Oct 30 03:42:52.102794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:52.102771    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4" event={"ID":"889e3849-cc76-4b08-9a6c-d038b4e0fe5b","Type":"ContainerStarted","Data":"2276d1ee7b4ebc0ee5560e2c737f3b7e83e52366ff115224fb2301fac3c9b735"}
Oct 30 03:42:52.102986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:52.102806    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:42:52.147899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:52.147834    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4" podStartSLOduration=1.147814735 podStartE2EDuration="1.147814735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:52.141949834 +0000 UTC m=+2400.570839251" watchObservedRunningTime="2026-10-30 03:42:52.147814735 +0000 UTC m=+2400.576704154"
Oct 30 03:42:52.332097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:52.332057    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:42:52.337256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:52.337230    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:42:56.816794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:42:56.816754    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"]
Oct 30 03:43:01.817673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:01.817618    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4" podUID="2b5a11a1-e0b6-41a0-997a-b76c9e968674" containerName="payload-processing" containerID="cri-o://22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488" gracePeriod=25
Oct 30 03:43:02.104303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.104269    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:43:02.170188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.170149    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume\") pod \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") "
Oct 30 03:43:02.170392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.170256    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w\") pod \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") "
Oct 30 03:43:02.170392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.170303    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume\") pod \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") "
Oct 30 03:43:02.170392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.170342    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials\") pod \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\" (UID: \"2b5a11a1-e0b6-41a0-997a-b76c9e968674\") "
Oct 30 03:43:02.170722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.170615    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume" pod "2b5a11a1-e0b6-41a0-997a-b76c9e968674" (UID: "2b5a11a1-e0b6-41a0-997a-b76c9e968674"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:02.171024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.170998    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume" pod "2b5a11a1-e0b6-41a0-997a-b76c9e968674" (UID: "2b5a11a1-e0b6-41a0-997a-b76c9e968674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:02.171159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.171135    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.171159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.171157    4877 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a11a1-e0b6-41a0-997a-b76c9e968674-routing-overlay-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.172811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.172787    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w" pod "2b5a11a1-e0b6-41a0-997a-b76c9e968674" (UID: "2b5a11a1-e0b6-41a0-997a-b76c9e968674"). InnerVolumeSpecName "kube-api-access-4495w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:02.173121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.173099    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials" pod "2b5a11a1-e0b6-41a0-997a-b76c9e968674" (UID: "2b5a11a1-e0b6-41a0-997a-b76c9e968674"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:02.201339 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.201301    4877 generic.go:356] "Generic (PLEG): container finished" podID="2b5a11a1-e0b6-41a0-997a-b76c9e968674" containerID="22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488" exitCode=0
Oct 30 03:43:02.201515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.201410    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"
Oct 30 03:43:02.201594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.201549    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4" event={"ID":"2b5a11a1-e0b6-41a0-997a-b76c9e968674","Type":"ContainerDied","Data":"22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488"}
Oct 30 03:43:02.201660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.201594    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4" event={"ID":"2b5a11a1-e0b6-41a0-997a-b76c9e968674","Type":"ContainerDied","Data":"3da89ac8dd7babab5070d530304d336f66b42167d5c17f4bc291ec2bba3f95b9"}
Oct 30 03:43:02.201660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.201621    4877 scope.go:122] "RemoveContainer" containerID="22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488"
Oct 30 03:43:02.216555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.216488    4877 scope.go:122] "RemoveContainer" containerID="22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488"
Oct 30 03:43:02.232450 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:43:02.230206    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488\": container with ID starting with 22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488 not found: ID does not exist" containerID="22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488"
Oct 30 03:43:02.232450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.230252    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488"} err="failed to get container status \"22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488\": rpc error: code = NotFound desc = could not find container \"22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488\": container with ID starting with 22816b42fa23110e9467c35bfbd8c9809fcb068493f9aab45ac90b552fd0f488 not found: ID does not exist"
Oct 30 03:43:02.256276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.256240    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"]
Oct 30 03:43:02.260452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.260410    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-557cqcrk4"]
Oct 30 03:43:02.272336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.272297    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4495w\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-kube-api-access-4495w\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:43:02.272336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:02.272332    4877 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2b5a11a1-e0b6-41a0-997a-b76c9e968674-provider-credentials\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.115188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:03.115158    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:43:03.702008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:03.701966    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5a11a1-e0b6-41a0-997a-b76c9e968674" path="/var/lib/kubelet/pods/2b5a11a1-e0b6-41a0-997a-b76c9e968674/volumes"
Oct 30 03:43:12.537912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.537874    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"]
Oct 30 03:43:12.538754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.538736    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2b5a11a1-e0b6-41a0-997a-b76c9e968674" containerName="payload-processing"
Oct 30 03:43:12.538805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.538755    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5a11a1-e0b6-41a0-997a-b76c9e968674" containerName="payload-processing"
Oct 30 03:43:12.538903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.538894    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5a11a1-e0b6-41a0-997a-b76c9e968674" containerName="payload-processing"
Oct 30 03:43:12.542804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.542784    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.558298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.558269    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"]
Oct 30 03:43:12.565205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.565145    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"e2e-cmp-legacy-e4ad0b-provider-cred\""
Oct 30 03:43:12.565344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.565316    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"openshift-service-ca.crt\""
Oct 30 03:43:12.565586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.565563    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-e2e--22d06fb9-dockercfg-chq67\""
Oct 30 03:43:12.565719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.565624    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689bcf\""
Oct 30 03:43:12.568508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.568475    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"kube-root-ca.crt\""
Oct 30 03:43:12.568508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.568488    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"routing-overlay\""
Oct 30 03:43:12.660928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.660890    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.661249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.661225    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.661374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.661360    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.661587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.661567    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkhp\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.772452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.770682    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.772452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.770765    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.772452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.770803    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.772452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.770920    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkhp\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.772452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.772332    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.777447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.772985    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.823246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.823163    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkhp\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.837468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.837180    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:12.858622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:12.858598    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:13.245458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:13.244550    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"]
Oct 30 03:43:13.355140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:13.355102    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj" event={"ID":"86d3867e-15c7-4148-869f-d6032c4342a4","Type":"ContainerStarted","Data":"bb273fdd1cdd4cb62c714ea0f1ede0f9e9aec86f38bfa0e3292e5be6dcd78362"}
Oct 30 03:43:14.095685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.095646    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"]
Oct 30 03:43:14.116932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.116901    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.122523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.122493    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"]
Oct 30 03:43:14.123239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.123103    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.123492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.123472    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.126811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.126786    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw584\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.126968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.126948    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.127113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.127095    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.127449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.127398    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.127625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.127605    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.127778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.127761    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.127936 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.127914    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229137    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229202    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229238    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw584\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229263    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229305    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229449    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229495    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229545    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229584    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.229678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.229624    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.230084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.230059    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.230216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.230188    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.230351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.230327    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.230580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.230455    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.231918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.231898    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.232120 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.232098    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.248244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.248212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.248473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.248446    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw584\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.367415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.367321    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj" event={"ID":"86d3867e-15c7-4148-869f-d6032c4342a4","Type":"ContainerStarted","Data":"ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527"}
Oct 30 03:43:14.367415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.367396    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:14.423450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.422110    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj" podStartSLOduration=2.422092246 podStartE2EDuration="2.422092246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:14.421668334 +0000 UTC m=+2422.850557753" watchObservedRunningTime="2026-10-30 03:43:14.422092246 +0000 UTC m=+2422.850981665"
Oct 30 03:43:14.445354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.444534    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:14.817734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.817382    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"]
Oct 30 03:43:14.823051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.823008    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:14.823168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.823102    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:14.823168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:14.823146    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:15.428446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:15.423506    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" event={"ID":"182b3d24-bb3c-41d1-bd43-acccb34ac751","Type":"ContainerStarted","Data":"ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d"}
Oct 30 03:43:15.428446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:15.423647    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" event={"ID":"182b3d24-bb3c-41d1-bd43-acccb34ac751","Type":"ContainerStarted","Data":"b132738cf6db6549290e35a3a57dbfc786687cdeeac73fac94f903212bb73033"}
Oct 30 03:43:15.445171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:15.445140    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:15.446573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:15.446545    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:15.446680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:15.446605    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:15.516785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:15.516712    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podStartSLOduration=1.51669002 podStartE2EDuration="1.51669002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:15.513560195 +0000 UTC m=+2423.942449633" watchObservedRunningTime="2026-10-30 03:43:15.51669002 +0000 UTC m=+2423.945579432"
Oct 30 03:43:16.286739 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:43:16.286659    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:43:16.456499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:16.447478    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:16.456499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:16.447545    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:17.462450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:17.459762    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:17.462450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:17.459833    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:17.502555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:17.502515    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:43:17.502837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:17.502807    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:43:18.446176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:18.445546    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:18.446176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:18.445597    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:19.445050 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:19.444950    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:19.445557 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:19.445040    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:20.445839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:20.445800    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:20.446478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:20.445868    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:21.445328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.445218    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 03:43:21.445328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.445288    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 03:43:21.622737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.622649    4877 generic.go:356] "Generic (PLEG): container finished" podID="ed33a285-7366-4359-93ff-1d4118773ab3" containerID="1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277" exitCode=0
Oct 30 03:43:21.623224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.622729    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" event={"ID":"ed33a285-7366-4359-93ff-1d4118773ab3","Type":"ContainerDied","Data":"1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"}
Oct 30 03:43:21.623334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.623316    4877 scope.go:122] "RemoveContainer" containerID="1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"
Oct 30 03:43:21.625483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.625454    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:21.625619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.625528    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:21.625619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:21.625559    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:22.516873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:22.516836    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:22.518582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:22.518555    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:22.519740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:22.519716    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:43:22.652698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:22.644983    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" event={"ID":"ed33a285-7366-4359-93ff-1d4118773ab3","Type":"ContainerStarted","Data":"1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa"}
Oct 30 03:43:23.170178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:23.170143    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:43:23.170544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:23.170519    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 03:43:23.170607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:23.170574    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 03:43:24.169581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.169541    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 03:43:24.171915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.169598    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 03:43:24.251563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.251524    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": read tcp 10.133.0.2:48528->10.133.0.199:15021: read: connection reset by peer" start-of-body=
Oct 30 03:43:24.252152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.251777    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": read tcp 10.133.0.2:48528->10.133.0.199:15021: read: connection reset by peer"
Oct 30 03:43:24.666036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.665994    4877 generic.go:356] "Generic (PLEG): container finished" podID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerID="7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083" exitCode=0
Oct 30 03:43:24.666219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.666068    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" event={"ID":"dd1b44f8-0fcb-4f16-85e1-87698900b711","Type":"ContainerDied","Data":"7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"}
Oct 30 03:43:24.666649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.666630    4877 scope.go:122] "RemoveContainer" containerID="7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"
Oct 30 03:43:24.668528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.668493    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:24.668616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.668582    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:24.668657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:24.668630    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:25.169921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:25.169887    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 03:43:25.170338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:25.169944    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 03:43:25.408737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:25.408701    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:43:25.697448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:25.681063    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" event={"ID":"dd1b44f8-0fcb-4f16-85e1-87698900b711","Type":"ContainerStarted","Data":"b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254"}
Oct 30 03:43:26.169492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:26.169458    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 03:43:26.169679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:26.169517    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 03:43:26.532496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:26.530828    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:43:26.532496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:26.530959    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 03:43:26.532496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:26.531018    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 03:43:27.173382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:27.169885    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 03:43:27.173382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:27.169943    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 03:43:27.538774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:27.531806    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 03:43:27.538774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:27.531867    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 03:43:28.170173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:28.170135    4877 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 03:43:28.170362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:28.170190    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 03:43:28.528579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:28.528544    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 03:43:28.528775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:28.528603    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 03:43:29.175042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:29.175010    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:43:29.175516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:29.175503    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:43:29.176139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:29.176122    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:43:29.535027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:29.533577    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 03:43:29.535027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:29.533651    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 03:43:30.528925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:30.528884    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 03:43:30.529350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:30.528955    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 03:43:31.529017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.528978    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 03:43:31.529526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.529046    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 03:43:31.935417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.935324    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"]
Oct 30 03:43:31.940392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.940372    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:31.956915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.956891    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 03:43:31.957151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.957114    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 03:43:31.957276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.957123    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 03:43:31.957276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.957199    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ttlms\""
Oct 30 03:43:31.975501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:31.975466    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"]
Oct 30 03:43:32.008911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.008860    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.009113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.009013    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94cq\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.009252 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.009224    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.009389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.009368    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.009513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.009462    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.009513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.009495    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.009643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.009543    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111029    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111120    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111157    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111174    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111310    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111406    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.111560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.111497    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94cq\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.112238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.112210    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.114315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.114281    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.114480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.114281    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.114480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.114373    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.114747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.114724    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.129748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.129726    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.129748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.129744    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94cq\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgmcn\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.257129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.256709    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.535750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.535721    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:43:32.536287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.536261    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:43:32.536980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.536952    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 03:43:32.722713 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:43:32.722665    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aa5563_67eb_4cbd_abe6_af26702f0252.slice/crio-40f6be7413474125ca0896f309011366e1c7439f9113660d92272829ee84100e WatchSource:0}: Error finding container 40f6be7413474125ca0896f309011366e1c7439f9113660d92272829ee84100e: Status 404 returned error can't find the container with id 40f6be7413474125ca0896f309011366e1c7439f9113660d92272829ee84100e
Oct 30 03:43:32.724596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.724568    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"]
Oct 30 03:43:32.726594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.726551    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:32.726707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.726642    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:43:32.886458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.886376    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" event={"ID":"04aa5563-67eb-4cbd-abe6-af26702f0252","Type":"ContainerStarted","Data":"704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c"}
Oct 30 03:43:32.886648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.886513    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" event={"ID":"04aa5563-67eb-4cbd-abe6-af26702f0252","Type":"ContainerStarted","Data":"40f6be7413474125ca0896f309011366e1c7439f9113660d92272829ee84100e"}
Oct 30 03:43:32.886648 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.886590    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:32.931564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:32.931494    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" podStartSLOduration=1.931473775 podStartE2EDuration="1.931473775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:32.927882007 +0000 UTC m=+2441.356771426" watchObservedRunningTime="2026-10-30 03:43:32.931473775 +0000 UTC m=+2441.360363246"
Oct 30 03:43:33.899634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:33.899554    4877 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-pgmcn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:43:33.899634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:33.899615    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" podUID="04aa5563-67eb-4cbd-abe6-af26702f0252" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:43:34.911098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:34.910676    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 03:43:45.030643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:45.030603    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"]
Oct 30 03:43:45.042131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:45.042104    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52120ea8-krcc8"]
Oct 30 03:43:45.703301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:45.703264    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c66b214-00e3-4863-94a2-4c12e0341221" path="/var/lib/kubelet/pods/0c66b214-00e3-4863-94a2-4c12e0341221/volumes"
Oct 30 03:43:46.230229 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:43:46.230191    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:43:56.029973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:56.029937    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"]
Oct 30 03:43:56.041701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:56.041670    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a12324-kpsv5"]
Oct 30 03:43:57.700870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:43:57.700828    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb67331-e1f7-4def-ba47-f4abd71c21d3" path="/var/lib/kubelet/pods/6fb67331-e1f7-4def-ba47-f4abd71c21d3/volumes"
Oct 30 03:44:10.029066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:10.028991    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"]
Oct 30 03:44:10.039996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:10.039968    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ec39aa-vhhrl"]
Oct 30 03:44:11.702011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:11.701969    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3" path="/var/lib/kubelet/pods/9871fe07-f664-4d0f-ad8f-9fbb54d6f3a3/volumes"
Oct 30 03:44:16.231143 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:44:16.231103    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:44:20.933771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:20.932850    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"]
Oct 30 03:44:21.071739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:21.070025    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"]
Oct 30 03:44:21.082684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:21.074210    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0acca6-5psn9"]
Oct 30 03:44:21.703781 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:21.703742    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba0babe-489f-4ba7-89ca-4aaf347a32cc" path="/var/lib/kubelet/pods/fba0babe-489f-4ba7-89ca-4aaf347a32cc/volumes"
Oct 30 03:44:23.174359 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.174318    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"]
Oct 30 03:44:23.179184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.179159    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.199012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.198984    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"]
Oct 30 03:44:23.206777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.204719    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:44:23.206777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.204790    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:44:23.353186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.353149    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788c2\" (UniqueName: \"kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.353380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.353271    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.353380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.353349    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.454994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.454897    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788c2\" (UniqueName: \"kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.454994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.454949    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.455355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.454998    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.456164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.455818    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.469729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.458074    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.501445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.499560    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788c2\" (UniqueName: \"kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2\") pod \"maas-api-revoke-sub-021e6dfc-jvb6b\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.508515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.508102    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:23.820578 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:44:23.819828    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84813703_5b47_472b_b87c_6eb3045bdcab.slice/crio-5b7edeb3cb54aca8cbbc03c96ef65fac299ee184de85246375f165cbf395b407 WatchSource:0}: Error finding container 5b7edeb3cb54aca8cbbc03c96ef65fac299ee184de85246375f165cbf395b407: Status 404 returned error can't find the container with id 5b7edeb3cb54aca8cbbc03c96ef65fac299ee184de85246375f165cbf395b407
Oct 30 03:44:23.823506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:23.823481    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"]
Oct 30 03:44:24.532484 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:24.532438    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b" event={"ID":"84813703-5b47-472b-b87c-6eb3045bdcab","Type":"ContainerStarted","Data":"1e86162a951a6ea06d22a7ece3a23257ef4236283bdfdcbea826f72deb8dfd55"}
Oct 30 03:44:24.532961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:24.532545    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b" event={"ID":"84813703-5b47-472b-b87c-6eb3045bdcab","Type":"ContainerStarted","Data":"5b7edeb3cb54aca8cbbc03c96ef65fac299ee184de85246375f165cbf395b407"}
Oct 30 03:44:24.570823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:24.570762    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b" podStartSLOduration=1.570748065 podStartE2EDuration="1.570748065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:24.568645091 +0000 UTC m=+2492.997534511" watchObservedRunningTime="2026-10-30 03:44:24.570748065 +0000 UTC m=+2492.999637494"
Oct 30 03:44:25.541820 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:25.541786    4877 generic.go:356] "Generic (PLEG): container finished" podID="84813703-5b47-472b-b87c-6eb3045bdcab" containerID="1e86162a951a6ea06d22a7ece3a23257ef4236283bdfdcbea826f72deb8dfd55" exitCode=0
Oct 30 03:44:25.542281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:25.541872    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b" event={"ID":"84813703-5b47-472b-b87c-6eb3045bdcab","Type":"ContainerDied","Data":"1e86162a951a6ea06d22a7ece3a23257ef4236283bdfdcbea826f72deb8dfd55"}
Oct 30 03:44:25.933900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:25.933814    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4" podUID="889e3849-cc76-4b08-9a6c-d038b4e0fe5b" containerName="payload-processing" containerID="cri-o://a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629" gracePeriod=25
Oct 30 03:44:26.244286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.244254    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:44:26.306891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.306849    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume\") pod \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") "
Oct 30 03:44:26.307093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.307041    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7\") pod \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") "
Oct 30 03:44:26.307178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.307128    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials\") pod \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") "
Oct 30 03:44:26.307249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.307193    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume\") pod \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\" (UID: \"889e3849-cc76-4b08-9a6c-d038b4e0fe5b\") "
Oct 30 03:44:26.307367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.307337    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume" pod "889e3849-cc76-4b08-9a6c-d038b4e0fe5b" (UID: "889e3849-cc76-4b08-9a6c-d038b4e0fe5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:26.307857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.307811    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:26.308204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.308156    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume" pod "889e3849-cc76-4b08-9a6c-d038b4e0fe5b" (UID: "889e3849-cc76-4b08-9a6c-d038b4e0fe5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:26.310940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.310910    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7" pod "889e3849-cc76-4b08-9a6c-d038b4e0fe5b" (UID: "889e3849-cc76-4b08-9a6c-d038b4e0fe5b"). InnerVolumeSpecName "kube-api-access-b56t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:26.317605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.317574    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials" pod "889e3849-cc76-4b08-9a6c-d038b4e0fe5b" (UID: "889e3849-cc76-4b08-9a6c-d038b4e0fe5b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:26.409687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.409653    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56t7\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-kube-api-access-b56t7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:26.409687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.409691    4877 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-provider-credentials\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:26.409934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.409719    4877 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/889e3849-cc76-4b08-9a6c-d038b4e0fe5b-routing-overlay-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:26.551533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.551499    4877 generic.go:356] "Generic (PLEG): container finished" podID="889e3849-cc76-4b08-9a6c-d038b4e0fe5b" containerID="a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629" exitCode=0
Oct 30 03:44:26.552034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.551551    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"
Oct 30 03:44:26.552034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.551587    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4" event={"ID":"889e3849-cc76-4b08-9a6c-d038b4e0fe5b","Type":"ContainerDied","Data":"a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629"}
Oct 30 03:44:26.552034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.551646    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4" event={"ID":"889e3849-cc76-4b08-9a6c-d038b4e0fe5b","Type":"ContainerDied","Data":"2276d1ee7b4ebc0ee5560e2c737f3b7e83e52366ff115224fb2301fac3c9b735"}
Oct 30 03:44:26.552034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.551670    4877 scope.go:122] "RemoveContainer" containerID="a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629"
Oct 30 03:44:26.571337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.571204    4877 scope.go:122] "RemoveContainer" containerID="a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629"
Oct 30 03:44:26.589896 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:44:26.585405    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629\": container with ID starting with a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629 not found: ID does not exist" containerID="a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629"
Oct 30 03:44:26.589896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.585467    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629"} err="failed to get container status \"a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629\": rpc error: code = NotFound desc = could not find container \"a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629\": container with ID starting with a463eb5fe011cb28b8971072ce079a740a6f1167c6f36db4b71af90415c4f629 not found: ID does not exist"
Oct 30 03:44:26.617486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.616048    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"]
Oct 30 03:44:26.621102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.621061    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-558769hv4"]
Oct 30 03:44:26.750521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.750497    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:26.816939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.816779    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2\" (UniqueName: \"kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2\") pod \"84813703-5b47-472b-b87c-6eb3045bdcab\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") "
Oct 30 03:44:26.817125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.817023    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca\") pod \"84813703-5b47-472b-b87c-6eb3045bdcab\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") "
Oct 30 03:44:26.817362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.817330    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca\") pod \"84813703-5b47-472b-b87c-6eb3045bdcab\" (UID: \"84813703-5b47-472b-b87c-6eb3045bdcab\") "
Oct 30 03:44:26.817849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.817818    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca" pod "84813703-5b47-472b-b87c-6eb3045bdcab" (UID: "84813703-5b47-472b-b87c-6eb3045bdcab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:26.818316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.818294    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:26.819760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.819736    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca" pod "84813703-5b47-472b-b87c-6eb3045bdcab" (UID: "84813703-5b47-472b-b87c-6eb3045bdcab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:26.820060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.820039    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2" pod "84813703-5b47-472b-b87c-6eb3045bdcab" (UID: "84813703-5b47-472b-b87c-6eb3045bdcab"). InnerVolumeSpecName "kube-api-access-788c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:26.919933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.919883    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788c2\" (UniqueName: \"kubernetes.io/projected/84813703-5b47-472b-b87c-6eb3045bdcab-kube-api-access-788c2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:26.919933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:26.919930    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84813703-5b47-472b-b87c-6eb3045bdcab-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:27.560988 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:27.560953    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"
Oct 30 03:44:27.561404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:27.560997    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b" event={"ID":"84813703-5b47-472b-b87c-6eb3045bdcab","Type":"ContainerDied","Data":"5b7edeb3cb54aca8cbbc03c96ef65fac299ee184de85246375f165cbf395b407"}
Oct 30 03:44:27.561404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:27.561055    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b7edeb3cb54aca8cbbc03c96ef65fac299ee184de85246375f165cbf395b407"
Oct 30 03:44:27.702626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:27.702579    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889e3849-cc76-4b08-9a6c-d038b4e0fe5b" path="/var/lib/kubelet/pods/889e3849-cc76-4b08-9a6c-d038b4e0fe5b/volumes"
Oct 30 03:44:34.236933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.236890    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"]
Oct 30 03:44:34.237666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.237650    4877 cpu_manager.go:414] "containerMap: removing container" podUID="889e3849-cc76-4b08-9a6c-d038b4e0fe5b" containerName="payload-processing"
Oct 30 03:44:34.237666 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.237667    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889e3849-cc76-4b08-9a6c-d038b4e0fe5b" containerName="payload-processing"
Oct 30 03:44:34.237784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.237708    4877 cpu_manager.go:414] "containerMap: removing container" podUID="84813703-5b47-472b-b87c-6eb3045bdcab" containerName="revoke-keys"
Oct 30 03:44:34.237784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.237713    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84813703-5b47-472b-b87c-6eb3045bdcab" containerName="revoke-keys"
Oct 30 03:44:34.237850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.237826    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="889e3849-cc76-4b08-9a6c-d038b4e0fe5b" containerName="payload-processing"
Oct 30 03:44:34.237850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.237840    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="84813703-5b47-472b-b87c-6eb3045bdcab" containerName="revoke-keys"
Oct 30 03:44:34.242646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.242625    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.262085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.262058    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"]
Oct 30 03:44:34.265703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.265678    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:44:34.267925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.267902    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:44:34.329244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.329208    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.329453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.329378    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.329572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.329552    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx28v\" (UniqueName: \"kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.431535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.431501    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx28v\" (UniqueName: \"kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.431737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.431620    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.431737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.431694    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.432365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.432338    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.434346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.434327    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.451832 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.451802    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx28v\" (UniqueName: \"kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v\") pod \"maas-api-revoke-sub-729bdfcf-g5qcm\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.556490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.556448    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:34.963251 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:44:34.963211    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2e3226_0cd4_45a8_a7ff_ee904a0e8d4b.slice/crio-8255eb2c1894447db3f6b6f29a660da2f64aa79cad34232feedb846663f5f38e WatchSource:0}: Error finding container 8255eb2c1894447db3f6b6f29a660da2f64aa79cad34232feedb846663f5f38e: Status 404 returned error can't find the container with id 8255eb2c1894447db3f6b6f29a660da2f64aa79cad34232feedb846663f5f38e
Oct 30 03:44:34.967076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:34.967045    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"]
Oct 30 03:44:35.502267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.502177    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"]
Oct 30 03:44:35.506776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.506755    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.514454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.514391    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpqt\" (UniqueName: \"kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.514560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.514497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.514602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.514571    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.525197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.525173    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"]
Oct 30 03:44:35.616039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.615998    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpqt\" (UniqueName: \"kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.616257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.616078    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.616257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.616158    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.617138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.617113    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.620127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.620097    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.641903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.641875    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpqt\" (UniqueName: \"kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt\") pod \"maas-api-revoke-sub-611f8e44-dqp77\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.827491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.827454    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:35.860341 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.860236    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm" event={"ID":"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b","Type":"ContainerStarted","Data":"78cca1a9e4e362243be15b98b95878eb8dd807f18ee121eda8f7ee5ad87f7694"}
Oct 30 03:44:35.860341 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.860308    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm" event={"ID":"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b","Type":"ContainerStarted","Data":"8255eb2c1894447db3f6b6f29a660da2f64aa79cad34232feedb846663f5f38e"}
Oct 30 03:44:35.906780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:35.906632    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm" podStartSLOduration=1.906613316 podStartE2EDuration="1.906613316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:35.90232972 +0000 UTC m=+2504.331219139" watchObservedRunningTime="2026-10-30 03:44:35.906613316 +0000 UTC m=+2504.335502734"
Oct 30 03:44:36.139856 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:44:36.139820    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514cb366_8797_4b9e_82d7_c25ed094c45c.slice/crio-495cfadb05986c8924e9697373a8d35d62e2026a0ca95d4ad898f716d353a910 WatchSource:0}: Error finding container 495cfadb05986c8924e9697373a8d35d62e2026a0ca95d4ad898f716d353a910: Status 404 returned error can't find the container with id 495cfadb05986c8924e9697373a8d35d62e2026a0ca95d4ad898f716d353a910
Oct 30 03:44:36.142007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:36.141981    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"]
Oct 30 03:44:36.902870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:36.902827    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77" event={"ID":"514cb366-8797-4b9e-82d7-c25ed094c45c","Type":"ContainerStarted","Data":"0273499e3967a6c10e5ce39084721d44dded6b97c94834029b0210b32f083ac9"}
Oct 30 03:44:36.903382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:36.902896    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77" event={"ID":"514cb366-8797-4b9e-82d7-c25ed094c45c","Type":"ContainerStarted","Data":"495cfadb05986c8924e9697373a8d35d62e2026a0ca95d4ad898f716d353a910"}
Oct 30 03:44:36.906501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:36.906462    4877 generic.go:356] "Generic (PLEG): container finished" podID="eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" containerID="78cca1a9e4e362243be15b98b95878eb8dd807f18ee121eda8f7ee5ad87f7694" exitCode=0
Oct 30 03:44:36.906749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:36.906652    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm" event={"ID":"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b","Type":"ContainerDied","Data":"78cca1a9e4e362243be15b98b95878eb8dd807f18ee121eda8f7ee5ad87f7694"}
Oct 30 03:44:37.002447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:37.001924    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77" podStartSLOduration=2.001903672 podStartE2EDuration="2.001903672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:36.955579984 +0000 UTC m=+2505.384469476" watchObservedRunningTime="2026-10-30 03:44:37.001903672 +0000 UTC m=+2505.430793091"
Oct 30 03:44:37.919399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:37.919366    4877 generic.go:356] "Generic (PLEG): container finished" podID="514cb366-8797-4b9e-82d7-c25ed094c45c" containerID="0273499e3967a6c10e5ce39084721d44dded6b97c94834029b0210b32f083ac9" exitCode=0
Oct 30 03:44:37.919893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:37.919469    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77" event={"ID":"514cb366-8797-4b9e-82d7-c25ed094c45c","Type":"ContainerDied","Data":"0273499e3967a6c10e5ce39084721d44dded6b97c94834029b0210b32f083ac9"}
Oct 30 03:44:38.117721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.117687    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:38.195131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.187951    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v\" (UniqueName: \"kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v\") pod \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") "
Oct 30 03:44:38.195131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.188170    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca\") pod \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") "
Oct 30 03:44:38.195131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.188321    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca\") pod \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\" (UID: \"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b\") "
Oct 30 03:44:38.201448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.200630    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca" pod "eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" (UID: "eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:38.236072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.231972    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v" pod "eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" (UID: "eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b"). InnerVolumeSpecName "kube-api-access-hx28v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:38.236072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.235805    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca" pod "eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" (UID: "eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:38.291137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.291095    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:38.291137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.291141    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:38.291379 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.291156    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx28v\" (UniqueName: \"kubernetes.io/projected/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b-kube-api-access-hx28v\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:38.931718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.931687    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"
Oct 30 03:44:38.932207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.931730    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm" event={"ID":"eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b","Type":"ContainerDied","Data":"8255eb2c1894447db3f6b6f29a660da2f64aa79cad34232feedb846663f5f38e"}
Oct 30 03:44:38.932207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:38.931801    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8255eb2c1894447db3f6b6f29a660da2f64aa79cad34232feedb846663f5f38e"
Oct 30 03:44:39.122235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.122204    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:39.208299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.208195    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca\") pod \"514cb366-8797-4b9e-82d7-c25ed094c45c\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") "
Oct 30 03:44:39.208481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.208405    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca\") pod \"514cb366-8797-4b9e-82d7-c25ed094c45c\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") "
Oct 30 03:44:39.208669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.208651    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt\" (UniqueName: \"kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt\") pod \"514cb366-8797-4b9e-82d7-c25ed094c45c\" (UID: \"514cb366-8797-4b9e-82d7-c25ed094c45c\") "
Oct 30 03:44:39.209054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.209024    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca" pod "514cb366-8797-4b9e-82d7-c25ed094c45c" (UID: "514cb366-8797-4b9e-82d7-c25ed094c45c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:39.209249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.209229    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.211117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.211077    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca" pod "514cb366-8797-4b9e-82d7-c25ed094c45c" (UID: "514cb366-8797-4b9e-82d7-c25ed094c45c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:39.212833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.212810    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt" pod "514cb366-8797-4b9e-82d7-c25ed094c45c" (UID: "514cb366-8797-4b9e-82d7-c25ed094c45c"). InnerVolumeSpecName "kube-api-access-ldpqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:39.310591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.310549    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldpqt\" (UniqueName: \"kubernetes.io/projected/514cb366-8797-4b9e-82d7-c25ed094c45c-kube-api-access-ldpqt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.310591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.310582    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/514cb366-8797-4b9e-82d7-c25ed094c45c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.909939 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:44:39.909906    4877 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514cb366_8797_4b9e_82d7_c25ed094c45c.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 03:44:39.940748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.940711    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77" event={"ID":"514cb366-8797-4b9e-82d7-c25ed094c45c","Type":"ContainerDied","Data":"495cfadb05986c8924e9697373a8d35d62e2026a0ca95d4ad898f716d353a910"}
Oct 30 03:44:39.941142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.940769    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="495cfadb05986c8924e9697373a8d35d62e2026a0ca95d4ad898f716d353a910"
Oct 30 03:44:39.941142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:39.940718    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"
Oct 30 03:44:45.393810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:45.393777    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"]
Oct 30 03:44:46.233493 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:44:46.233453    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:44:50.394189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.394138    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj" podUID="86d3867e-15c7-4148-869f-d6032c4342a4" containerName="payload-processing" containerID="cri-o://ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527" gracePeriod=25
Oct 30 03:44:50.694849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.694823    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:44:50.805470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.805406    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials\") pod \"86d3867e-15c7-4148-869f-d6032c4342a4\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") "
Oct 30 03:44:50.805676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.805501    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume\") pod \"86d3867e-15c7-4148-869f-d6032c4342a4\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") "
Oct 30 03:44:50.805676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.805653    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp\") pod \"86d3867e-15c7-4148-869f-d6032c4342a4\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") "
Oct 30 03:44:50.805809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.805754    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume\") pod \"86d3867e-15c7-4148-869f-d6032c4342a4\" (UID: \"86d3867e-15c7-4148-869f-d6032c4342a4\") "
Oct 30 03:44:50.806344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.806315    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume" pod "86d3867e-15c7-4148-869f-d6032c4342a4" (UID: "86d3867e-15c7-4148-869f-d6032c4342a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:50.806549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.806413    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume" pod "86d3867e-15c7-4148-869f-d6032c4342a4" (UID: "86d3867e-15c7-4148-869f-d6032c4342a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:50.808687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.808659    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp" pod "86d3867e-15c7-4148-869f-d6032c4342a4" (UID: "86d3867e-15c7-4148-869f-d6032c4342a4"). InnerVolumeSpecName "kube-api-access-6wkhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:50.808983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.808934    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials" pod "86d3867e-15c7-4148-869f-d6032c4342a4" (UID: "86d3867e-15c7-4148-869f-d6032c4342a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:50.913032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.911102    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wkhp\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-kube-api-access-6wkhp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:50.913032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.911149    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:50.913032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.911163    4877 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d3867e-15c7-4148-869f-d6032c4342a4-provider-credentials\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:50.913032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:50.911177    4877 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d3867e-15c7-4148-869f-d6032c4342a4-routing-overlay-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:44:51.126179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.126143    4877 generic.go:356] "Generic (PLEG): container finished" podID="86d3867e-15c7-4148-869f-d6032c4342a4" containerID="ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527" exitCode=0
Oct 30 03:44:51.126394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.126228    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"
Oct 30 03:44:51.126394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.126231    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj" event={"ID":"86d3867e-15c7-4148-869f-d6032c4342a4","Type":"ContainerDied","Data":"ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527"}
Oct 30 03:44:51.126394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.126383    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj" event={"ID":"86d3867e-15c7-4148-869f-d6032c4342a4","Type":"ContainerDied","Data":"bb273fdd1cdd4cb62c714ea0f1ede0f9e9aec86f38bfa0e3292e5be6dcd78362"}
Oct 30 03:44:51.126605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.126414    4877 scope.go:122] "RemoveContainer" containerID="ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527"
Oct 30 03:44:51.138800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.138773    4877 scope.go:122] "RemoveContainer" containerID="ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527"
Oct 30 03:44:51.139171 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:44:51.139149    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527\": container with ID starting with ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527 not found: ID does not exist" containerID="ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527"
Oct 30 03:44:51.139219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.139181    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527"} err="failed to get container status \"ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527\": rpc error: code = NotFound desc = could not find container \"ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527\": container with ID starting with ce1e68ad1da7a5edff2c32f8c00a50c22a46edb0174c283b74a9318f3593f527 not found: ID does not exist"
Oct 30 03:44:51.183063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.182974    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"]
Oct 30 03:44:51.195530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.195499    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-75c977lzj"]
Oct 30 03:44:51.700363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:51.700327    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d3867e-15c7-4148-869f-d6032c4342a4" path="/var/lib/kubelet/pods/86d3867e-15c7-4148-869f-d6032c4342a4/volumes"
Oct 30 03:44:55.138044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:55.138011    4877 scope.go:122] "RemoveContainer" containerID="b5f74ee87a088a4eafa6ce89bcfa44a91401adcadc36b259460ecfffc4c28676"
Oct 30 03:44:55.149285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:55.149267    4877 scope.go:122] "RemoveContainer" containerID="9afbf8bf7a3c629a4dd95ee284646e0c5ae3832b2e2c6b7b6682a73eb198ca56"
Oct 30 03:44:59.029017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:59.028968    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"]
Oct 30 03:44:59.040177 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:59.040136    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47467683-lwljc"]
Oct 30 03:44:59.700331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:44:59.700294    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cfcdea-e95b-4f48-9cdb-4f24eb57df0d" path="/var/lib/kubelet/pods/21cfcdea-e95b-4f48-9cdb-4f24eb57df0d/volumes"
Oct 30 03:45:00.144959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.144918    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"]
Oct 30 03:45:00.145837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.145815    4877 cpu_manager.go:414] "containerMap: removing container" podUID="514cb366-8797-4b9e-82d7-c25ed094c45c" containerName="revoke-keys"
Oct 30 03:45:00.145837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.145837    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514cb366-8797-4b9e-82d7-c25ed094c45c" containerName="revoke-keys"
Oct 30 03:45:00.146019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.145870    4877 cpu_manager.go:414] "containerMap: removing container" podUID="86d3867e-15c7-4148-869f-d6032c4342a4" containerName="payload-processing"
Oct 30 03:45:00.146019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.145888    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d3867e-15c7-4148-869f-d6032c4342a4" containerName="payload-processing"
Oct 30 03:45:00.146019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.145898    4877 cpu_manager.go:414] "containerMap: removing container" podUID="eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" containerName="revoke-keys"
Oct 30 03:45:00.146019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.145904    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" containerName="revoke-keys"
Oct 30 03:45:00.146229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.146046    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="514cb366-8797-4b9e-82d7-c25ed094c45c" containerName="revoke-keys"
Oct 30 03:45:00.146229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.146055    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d3867e-15c7-4148-869f-d6032c4342a4" containerName="payload-processing"
Oct 30 03:45:00.146229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.146062    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" containerName="revoke-keys"
Oct 30 03:45:00.151387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.151362    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"]
Oct 30 03:45:00.151544 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.151501    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.156459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.156413    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"]
Oct 30 03:45:00.156583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.156505    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.161329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.161300    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnrk\" (UniqueName: \"kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.161463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.161405    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.161552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.161531    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888865-57mfd\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.161615 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.161570    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6wg\" (UniqueName: \"kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg\") pod \"maas-api-key-cleanup-29888865-57mfd\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.161899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.161879    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"]
Oct 30 03:45:00.162413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.162393    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.166396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.166374    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"]
Oct 30 03:45:00.166522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.166508    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.171797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.171775    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"]
Oct 30 03:45:00.172002 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.171978    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.176378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.176351    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:45:00.176979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.176953    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"]
Oct 30 03:45:00.177091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.176988    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.205199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.205171    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"]
Oct 30 03:45:00.206396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.206226    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.212135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.212003    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"]
Oct 30 03:45:00.212460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.212411    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265054    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnrk\" (UniqueName: \"kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265141    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265197    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tp8\" (UniqueName: \"kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265248    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265280    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dj9h\" (UniqueName: \"kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265339    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265408    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7v7\" (UniqueName: \"kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265522    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888865-57mfd\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265557    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447ch\" (UniqueName: \"kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265588    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6wg\" (UniqueName: \"kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg\") pod \"maas-api-key-cleanup-29888865-57mfd\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265616    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxdw\" (UniqueName: \"kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265718    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265755    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnf7z\" (UniqueName: \"kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265802    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265881    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.265948    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.267133    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.287971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.268371    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888865-57mfd\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.289218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.288354    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnrk\" (UniqueName: \"kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.292851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.292806    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"]
Oct 30 03:45:00.293019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.292909    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.299690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.299658    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6wg\" (UniqueName: \"kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg\") pod \"maas-api-key-cleanup-29888865-57mfd\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.301485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.301355    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"]
Oct 30 03:45:00.301485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.301396    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"]
Oct 30 03:45:00.301485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.301463    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"]
Oct 30 03:45:00.301711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.301586    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.305926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.304329    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"]
Oct 30 03:45:00.309187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.309163    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"]
Oct 30 03:45:00.315036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.315017    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"]
Oct 30 03:45:00.315124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.315113    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.317567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.317547    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"]
Oct 30 03:45:00.324874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.324850    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"]
Oct 30 03:45:00.324978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.324951    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.328644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.328618    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"]
Oct 30 03:45:00.340916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.340889    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"]
Oct 30 03:45:00.346669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.346641    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.346976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.346961    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"]
Oct 30 03:45:00.358706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.358681    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"]
Oct 30 03:45:00.358873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.358729    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"]
Oct 30 03:45:00.358873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.358795    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.363802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.363781    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"]
Oct 30 03:45:00.367929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.367887    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.368083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.367959    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vk6f\" (UniqueName: \"kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.368083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368002    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.368083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368045    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.368083 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368073    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.368313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368101    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2h9\" (UniqueName: \"kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.368313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368155    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.368313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368188    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.368313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368239    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tp8\" (UniqueName: \"kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.368313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368285    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.368616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368313    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dj9h\" (UniqueName: \"kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.368616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368339    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.368616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368406    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsrc\" (UniqueName: \"kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.368616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368494    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7v7\" (UniqueName: \"kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.368616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368523    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hrt\" (UniqueName: \"kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.368616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368602    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.369080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368657    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.369080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368687    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447ch\" (UniqueName: \"kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.369080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368704    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.369080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368716    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxdw\" (UniqueName: \"kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.369080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368732    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.369080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.368748    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87nb5\" (UniqueName: \"kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.369456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.369303    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.369552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.369525    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqxg\" (UniqueName: \"kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.369661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.369602    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.369661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.369634    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnf7z\" (UniqueName: \"kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.369782 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.369665    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.370020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.369996    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.370349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.370291    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.370741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.370718    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.371192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.371170    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"]
Oct 30 03:45:00.378095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.378075    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"]
Oct 30 03:45:00.389485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.384408    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"]
Oct 30 03:45:00.390114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.390087    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"]
Oct 30 03:45:00.390217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.390201    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.392322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.392299    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447ch\" (UniqueName: \"kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.392601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.392576    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tp8\" (UniqueName: \"kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.395531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.395476    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dj9h\" (UniqueName: \"kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.395658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.395492    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxdw\" (UniqueName: \"kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.396891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.396864    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7v7\" (UniqueName: \"kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.397547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.397527    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnf7z\" (UniqueName: \"kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.398804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.398785    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"]
Oct 30 03:45:00.402851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.402832    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"]
Oct 30 03:45:00.470756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.470725    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.470928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.470774    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.470928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.470802    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87nb5\" (UniqueName: \"kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.470928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.470837    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqxg\" (UniqueName: \"kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.470928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.470897    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.471167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.470978    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vk6f\" (UniqueName: \"kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.471167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471029    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.471167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471062    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2h9\" (UniqueName: \"kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.471167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471103    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.471167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471145    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.471476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471223    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.471476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471274    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6h2\" (UniqueName: \"kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.471476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471312    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsrc\" (UniqueName: \"kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.471476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471399    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hrt\" (UniqueName: \"kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.471709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.471839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.471811    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:00.472296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.472269    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.473016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.472981    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.473184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.473156    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.474858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.474835    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.476526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.476501    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.495492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.495414    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsrc\" (UniqueName: \"kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.495977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.495957    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hrt\" (UniqueName: \"kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.497014 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.496988    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vk6f\" (UniqueName: \"kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.497153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.497139    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:00.498076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.498040    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2h9\" (UniqueName: \"kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.498225 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.498195    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87nb5\" (UniqueName: \"kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.499921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.499899    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqxg\" (UniqueName: \"kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.521112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.521081    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:00.540964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.538901    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:00.614718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.596539    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6h2\" (UniqueName: \"kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.614718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.596984    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.614718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.601842    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.614718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.602359    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:00.614718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.605714    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:00.639210 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.639072    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6h2\" (UniqueName: \"kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:00.640124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.639919    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:00.649679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.649388    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:00.668507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.665573    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:00.673673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.673639    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:00.686551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.686485    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:00.699051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.698991    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:00.781959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.781707    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:00.794797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.792358    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:00.841445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:00.825114    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:01.032564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.032459    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"]
Oct 30 03:45:01.045013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.044493    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"]
Oct 30 03:45:01.054819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.054785    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f58b94a-9gnnw"]
Oct 30 03:45:01.294528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.294492    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerStarted","Data":"9830863a66d9ba0b1fea519a149d5aa1c4a2c67f617a565a28a4a34f500d3584"}
Oct 30 03:45:01.296850 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:45:01.296813    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918ac658_590a_4014_aa4e_0cdb4a173213.slice/crio-3447c7e03aea979140e4fdd6f5242f07963fca5776d90e14d6db9564947803b2 WatchSource:0}: Error finding container 3447c7e03aea979140e4fdd6f5242f07963fca5776d90e14d6db9564947803b2: Status 404 returned error can't find the container with id 3447c7e03aea979140e4fdd6f5242f07963fca5776d90e14d6db9564947803b2
Oct 30 03:45:01.302872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.301515    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"]
Oct 30 03:45:01.310643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.310606    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"]
Oct 30 03:45:01.311614 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:45:01.311584    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe8de4b_ae72_49e7_8cc3_d21faa913e02.slice/crio-5cc71c95dbd24a051cc82f603404ca010c3ef54624916560915d667aee4f584a WatchSource:0}: Error finding container 5cc71c95dbd24a051cc82f603404ca010c3ef54624916560915d667aee4f584a: Status 404 returned error can't find the container with id 5cc71c95dbd24a051cc82f603404ca010c3ef54624916560915d667aee4f584a
Oct 30 03:45:01.507144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.507116    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"]
Oct 30 03:45:01.512769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.512727    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"]
Oct 30 03:45:01.517437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.517398    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"]
Oct 30 03:45:01.553829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.553786    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"]
Oct 30 03:45:01.558319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.558297    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"]
Oct 30 03:45:01.562943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.562920    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"]
Oct 30 03:45:01.612563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.612532    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"]
Oct 30 03:45:01.617641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.617614    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"]
Oct 30 03:45:01.649724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.649698    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"]
Oct 30 03:45:01.687161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.685219    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"]
Oct 30 03:45:01.709895 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:45:01.709870    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09029590_3cdc_4fa7_b239_79c5db99fa0e.slice/crio-7ca193c42dd720be445fca595272becd1f5c219d0797799fde0be24bd13dd275 WatchSource:0}: Error finding container 7ca193c42dd720be445fca595272becd1f5c219d0797799fde0be24bd13dd275: Status 404 returned error can't find the container with id 7ca193c42dd720be445fca595272becd1f5c219d0797799fde0be24bd13dd275
Oct 30 03:45:01.730235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.730208    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b9418b-70fc-42c7-8b72-3ca70d3f08b2" path="/var/lib/kubelet/pods/47b9418b-70fc-42c7-8b72-3ca70d3f08b2/volumes"
Oct 30 03:45:01.730884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.730857    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"]
Oct 30 03:45:01.730976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:01.730923    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"]
Oct 30 03:45:02.305842 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.305787    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerStarted","Data":"df37cfbc60c22568e3b675dcf2fe2ea19a0d395596441d0c5e793a72fa78264a"}
Oct 30 03:45:02.307487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.307455    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerStarted","Data":"44c85ccd181249e3b842c8456ff5ec323c9b039216fce3426ee7c65df27d7c7e"}
Oct 30 03:45:02.309037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.309012    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerStarted","Data":"d3cf7539d89fd35ed666b1ec5f4093c0c2516d90fb3824d70dd9ec14f97a5633"}
Oct 30 03:45:02.310459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.310395    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" event={"ID":"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8","Type":"ContainerStarted","Data":"347601d3375dacc839f860bd070b50cabfb4efee57c283e01517f84bb9c8b0db"}
Oct 30 03:45:02.312029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.312000    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerStarted","Data":"5cc71c95dbd24a051cc82f603404ca010c3ef54624916560915d667aee4f584a"}
Oct 30 03:45:02.313792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.313768    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerStarted","Data":"2f6ca5ce71126686d3a86bdcf64d7ad3671f6d0e9e41a790befc0e7d99c1b157"}
Oct 30 03:45:02.313916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.313814    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerStarted","Data":"3447c7e03aea979140e4fdd6f5242f07963fca5776d90e14d6db9564947803b2"}
Oct 30 03:45:02.315269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.315245    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerStarted","Data":"32a654497cadf6f69a4003ca76d9c6a0bd5bd77a1c4be63fc722a4ec0e65f2af"}
Oct 30 03:45:02.317158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.317133    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerStarted","Data":"a8f344b05f4835ef4a8b48baac189b5a9eada85a0c3f20e914b1d7600a1d15ce"}
Oct 30 03:45:02.318851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.318825    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerStarted","Data":"ce2c33aef87b7716b922fea62a397c08b78ab3889afd5b151068304de1461907"}
Oct 30 03:45:02.321114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.320359    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerStarted","Data":"3a9acc01531c447cb555975b1224710d224ba0957be187712f4866c2346d977c"}
Oct 30 03:45:02.328471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.328439    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerStarted","Data":"3472a4b83ba863e4c77a21a3ee8c36512c246898e89523d027f47e50b1a8c7ee"}
Oct 30 03:45:02.352358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.352301    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" event={"ID":"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739","Type":"ContainerStarted","Data":"9adb0ea5440e5e6b4aa8eff7d75bd7818e0e3f60c41f79aa43bdb7566c590695"}
Oct 30 03:45:02.355452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.355401    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerStarted","Data":"a2dd8a46c4db47bfe3ff14045b15884fa53ef986f2fcb852b6c17192a9e70ebe"}
Oct 30 03:45:02.359689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.359651    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerStarted","Data":"3e44a21684347f2b2ea863f83ca683c51bb64989bee91c454c6185d8f6b5b7dd"}
Oct 30 03:45:02.362007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.361958    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerStarted","Data":"7ca193c42dd720be445fca595272becd1f5c219d0797799fde0be24bd13dd275"}
Oct 30 03:45:02.388140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.388035    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" podStartSLOduration=2.3880143289999998 podStartE2EDuration="2.388014329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:02.382350255 +0000 UTC m=+2530.811239674" watchObservedRunningTime="2026-10-30 03:45:02.388014329 +0000 UTC m=+2530.816903750"
Oct 30 03:45:02.429124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:02.429025    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" podStartSLOduration=2.428998179 podStartE2EDuration="2.428998179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:02.428624752 +0000 UTC m=+2530.857514170" watchObservedRunningTime="2026-10-30 03:45:02.428998179 +0000 UTC m=+2530.857887598"
Oct 30 03:45:03.373925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.373883    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerStarted","Data":"0fd4b076b99a3a1f0918ba2d331f71235ab36d48cef9707075524642f197d637"}
Oct 30 03:45:03.375700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.375672    4877 generic.go:356] "Generic (PLEG): container finished" podID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerID="4233c4b210820d2e2443beed60f3496f1293b66b7c861aa66088aca057a979d8" exitCode=60
Oct 30 03:45:03.375859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.375751    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerDied","Data":"4233c4b210820d2e2443beed60f3496f1293b66b7c861aa66088aca057a979d8"}
Oct 30 03:45:03.376064 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.376045    4877 scope.go:122] "RemoveContainer" containerID="4233c4b210820d2e2443beed60f3496f1293b66b7c861aa66088aca057a979d8"
Oct 30 03:45:03.384363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.384326    4877 generic.go:356] "Generic (PLEG): container finished" podID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerID="a2c692f079837f09237ac2af32cce439de7e5a625ae707200832e1449beb94e7" exitCode=60
Oct 30 03:45:03.384541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.384486    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerDied","Data":"a2c692f079837f09237ac2af32cce439de7e5a625ae707200832e1449beb94e7"}
Oct 30 03:45:03.385027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.384996    4877 scope.go:122] "RemoveContainer" containerID="a2c692f079837f09237ac2af32cce439de7e5a625ae707200832e1449beb94e7"
Oct 30 03:45:03.395393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.395361    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerStarted","Data":"cd63df22fcb5cba409a561771b5b8af25559595f197ad4a0fae7bab33412d9a7"}
Oct 30 03:45:03.398316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.398291    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" event={"ID":"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8","Type":"ContainerStarted","Data":"e7d46a007246d2fb4b3bb40b3b80327b7e30af4d33f296ca4409c4faf772f79c"}
Oct 30 03:45:03.400197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.400175    4877 generic.go:356] "Generic (PLEG): container finished" podID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerID="9b22b00b104af2343e72b150dda10ebc2fb9fd0ca965f95a2e9106131ed78e8e" exitCode=60
Oct 30 03:45:03.400293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.400254    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerDied","Data":"9b22b00b104af2343e72b150dda10ebc2fb9fd0ca965f95a2e9106131ed78e8e"}
Oct 30 03:45:03.400530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.400509    4877 scope.go:122] "RemoveContainer" containerID="9b22b00b104af2343e72b150dda10ebc2fb9fd0ca965f95a2e9106131ed78e8e"
Oct 30 03:45:03.402122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.402097    4877 generic.go:356] "Generic (PLEG): container finished" podID="918ac658-590a-4014-aa4e-0cdb4a173213" containerID="2f6ca5ce71126686d3a86bdcf64d7ad3671f6d0e9e41a790befc0e7d99c1b157" exitCode=60
Oct 30 03:45:03.402229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.402184    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerDied","Data":"2f6ca5ce71126686d3a86bdcf64d7ad3671f6d0e9e41a790befc0e7d99c1b157"}
Oct 30 03:45:03.402404 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.402389    4877 scope.go:122] "RemoveContainer" containerID="2f6ca5ce71126686d3a86bdcf64d7ad3671f6d0e9e41a790befc0e7d99c1b157"
Oct 30 03:45:03.408474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.408405    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerStarted","Data":"a8827c0460442bf574d6c16ef36dd8ff746aa3d2f5e1cbae2e0abef6ff51c8d5"}
Oct 30 03:45:03.410855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.410830    4877 generic.go:356] "Generic (PLEG): container finished" podID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerID="146b4e1b51615d86edfdb4be5f7e9e24a34952e03cbf0a3257a2753ffec2533f" exitCode=60
Oct 30 03:45:03.410958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.410908    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerDied","Data":"146b4e1b51615d86edfdb4be5f7e9e24a34952e03cbf0a3257a2753ffec2533f"}
Oct 30 03:45:03.411130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.411115    4877 scope.go:122] "RemoveContainer" containerID="146b4e1b51615d86edfdb4be5f7e9e24a34952e03cbf0a3257a2753ffec2533f"
Oct 30 03:45:03.413101 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.413080    4877 generic.go:356] "Generic (PLEG): container finished" podID="a168bbc2-f646-40d7-897c-0531070a71a5" containerID="ec1a04768c652c87f7f630f7fe43475da3f678301bc75094c7568f2de6311b3f" exitCode=60
Oct 30 03:45:03.413228 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.413146    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerDied","Data":"ec1a04768c652c87f7f630f7fe43475da3f678301bc75094c7568f2de6311b3f"}
Oct 30 03:45:03.413410 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.413391    4877 scope.go:122] "RemoveContainer" containerID="ec1a04768c652c87f7f630f7fe43475da3f678301bc75094c7568f2de6311b3f"
Oct 30 03:45:03.415840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.415814    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerStarted","Data":"d1f8ad2d99bc9ab391b1e2ccbeab94597b38c988af642a858243072b5c2791ca"}
Oct 30 03:45:03.431521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.431494    4877 generic.go:356] "Generic (PLEG): container finished" podID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerID="3472a4b83ba863e4c77a21a3ee8c36512c246898e89523d027f47e50b1a8c7ee" exitCode=60
Oct 30 03:45:03.431686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.431668    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerDied","Data":"3472a4b83ba863e4c77a21a3ee8c36512c246898e89523d027f47e50b1a8c7ee"}
Oct 30 03:45:03.431783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.431740    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" podStartSLOduration=3.43172226 podStartE2EDuration="3.43172226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.427052128 +0000 UTC m=+2531.855941562" watchObservedRunningTime="2026-10-30 03:45:03.43172226 +0000 UTC m=+2531.860611679"
Oct 30 03:45:03.432313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.432106    4877 scope.go:122] "RemoveContainer" containerID="3472a4b83ba863e4c77a21a3ee8c36512c246898e89523d027f47e50b1a8c7ee"
Oct 30 03:45:03.435188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.435123    4877 generic.go:356] "Generic (PLEG): container finished" podID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerID="ce065c34dd652f422ddae7d6bc005b47687af849e65b9b70970e5e9bf0a02de1" exitCode=60
Oct 30 03:45:03.435368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.435170    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" event={"ID":"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739","Type":"ContainerDied","Data":"ce065c34dd652f422ddae7d6bc005b47687af849e65b9b70970e5e9bf0a02de1"}
Oct 30 03:45:03.435536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.435515    4877 scope.go:122] "RemoveContainer" containerID="ce065c34dd652f422ddae7d6bc005b47687af849e65b9b70970e5e9bf0a02de1"
Oct 30 03:45:03.439594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.439567    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerStarted","Data":"a1677e40cedba9258438e82d5b32161f53d3b94572873b7d130e88ef9be0858b"}
Oct 30 03:45:03.441607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.441585    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerStarted","Data":"e028e06ffd77c2db65247320d50070e35b53bd8916782a019ea7d42bb5b6da72"}
Oct 30 03:45:03.508137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.508054    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" podStartSLOduration=3.508040474 podStartE2EDuration="3.508040474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.507604218 +0000 UTC m=+2531.936493660" watchObservedRunningTime="2026-10-30 03:45:03.508040474 +0000 UTC m=+2531.936929892"
Oct 30 03:45:03.549135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.549089    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" podStartSLOduration=3.549074916 podStartE2EDuration="3.549074916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.544885147 +0000 UTC m=+2531.973774555" watchObservedRunningTime="2026-10-30 03:45:03.549074916 +0000 UTC m=+2531.977964334"
Oct 30 03:45:03.703213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.703085    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" podStartSLOduration=3.703063716 podStartE2EDuration="3.703063716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.694257461 +0000 UTC m=+2532.123146879" watchObservedRunningTime="2026-10-30 03:45:03.703063716 +0000 UTC m=+2532.131953136"
Oct 30 03:45:03.736381 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.736323    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" podStartSLOduration=3.73630824 podStartE2EDuration="3.73630824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.734859718 +0000 UTC m=+2532.163749158" watchObservedRunningTime="2026-10-30 03:45:03.73630824 +0000 UTC m=+2532.165197658"
Oct 30 03:45:03.883251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.883200    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" podStartSLOduration=3.883184815 podStartE2EDuration="3.883184815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.881908059 +0000 UTC m=+2532.310797499" watchObservedRunningTime="2026-10-30 03:45:03.883184815 +0000 UTC m=+2532.312074233"
Oct 30 03:45:03.957687 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:03.957563    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" podStartSLOduration=3.957547429 podStartE2EDuration="3.957547429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:03.956788208 +0000 UTC m=+2532.385677629" watchObservedRunningTime="2026-10-30 03:45:03.957547429 +0000 UTC m=+2532.386436848"
Oct 30 03:45:04.452498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.452458    4877 generic.go:356] "Generic (PLEG): container finished" podID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerID="0fd4b076b99a3a1f0918ba2d331f71235ab36d48cef9707075524642f197d637" exitCode=60
Oct 30 03:45:04.453013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.452550    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerDied","Data":"0fd4b076b99a3a1f0918ba2d331f71235ab36d48cef9707075524642f197d637"}
Oct 30 03:45:04.453013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.452907    4877 scope.go:122] "RemoveContainer" containerID="0fd4b076b99a3a1f0918ba2d331f71235ab36d48cef9707075524642f197d637"
Oct 30 03:45:04.455231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.455208    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerStarted","Data":"526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de"}
Oct 30 03:45:04.457468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.457446    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerStarted","Data":"9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4"}
Oct 30 03:45:04.459719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.459690    4877 generic.go:356] "Generic (PLEG): container finished" podID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerID="cd63df22fcb5cba409a561771b5b8af25559595f197ad4a0fae7bab33412d9a7" exitCode=60
Oct 30 03:45:04.459817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.459800    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerDied","Data":"cd63df22fcb5cba409a561771b5b8af25559595f197ad4a0fae7bab33412d9a7"}
Oct 30 03:45:04.460181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.460155    4877 scope.go:122] "RemoveContainer" containerID="cd63df22fcb5cba409a561771b5b8af25559595f197ad4a0fae7bab33412d9a7"
Oct 30 03:45:04.462572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.462546    4877 generic.go:356] "Generic (PLEG): container finished" podID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerID="e7d46a007246d2fb4b3bb40b3b80327b7e30af4d33f296ca4409c4faf772f79c" exitCode=60
Oct 30 03:45:04.462831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.462809    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" event={"ID":"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8","Type":"ContainerDied","Data":"e7d46a007246d2fb4b3bb40b3b80327b7e30af4d33f296ca4409c4faf772f79c"}
Oct 30 03:45:04.462958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.462939    4877 scope.go:122] "RemoveContainer" containerID="e7d46a007246d2fb4b3bb40b3b80327b7e30af4d33f296ca4409c4faf772f79c"
Oct 30 03:45:04.465801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.465408    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerStarted","Data":"7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a"}
Oct 30 03:45:04.473743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.473627    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerStarted","Data":"d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a"}
Oct 30 03:45:04.476317 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.476291    4877 generic.go:356] "Generic (PLEG): container finished" podID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerID="a8827c0460442bf574d6c16ef36dd8ff746aa3d2f5e1cbae2e0abef6ff51c8d5" exitCode=60
Oct 30 03:45:04.476413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.476343    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerDied","Data":"a8827c0460442bf574d6c16ef36dd8ff746aa3d2f5e1cbae2e0abef6ff51c8d5"}
Oct 30 03:45:04.479461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.476671    4877 scope.go:122] "RemoveContainer" containerID="a8827c0460442bf574d6c16ef36dd8ff746aa3d2f5e1cbae2e0abef6ff51c8d5"
Oct 30 03:45:04.479461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.479336    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerStarted","Data":"68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9"}
Oct 30 03:45:04.482467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.482402    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerStarted","Data":"ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272"}
Oct 30 03:45:04.703149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.694565    4877 generic.go:356] "Generic (PLEG): container finished" podID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerID="d1f8ad2d99bc9ab391b1e2ccbeab94597b38c988af642a858243072b5c2791ca" exitCode=60
Oct 30 03:45:04.703149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.694707    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerDied","Data":"d1f8ad2d99bc9ab391b1e2ccbeab94597b38c988af642a858243072b5c2791ca"}
Oct 30 03:45:04.703149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.695214    4877 scope.go:122] "RemoveContainer" containerID="d1f8ad2d99bc9ab391b1e2ccbeab94597b38c988af642a858243072b5c2791ca"
Oct 30 03:45:04.719787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.719099    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerStarted","Data":"3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1"}
Oct 30 03:45:04.754641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.754579    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" podStartSLOduration=4.754556449 podStartE2EDuration="4.754556449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:04.722872868 +0000 UTC m=+2533.151762286" watchObservedRunningTime="2026-10-30 03:45:04.754556449 +0000 UTC m=+2533.183445868"
Oct 30 03:45:04.763200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.762880    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" event={"ID":"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739","Type":"ContainerStarted","Data":"5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc"}
Oct 30 03:45:04.765293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.765268    4877 generic.go:356] "Generic (PLEG): container finished" podID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerID="a1677e40cedba9258438e82d5b32161f53d3b94572873b7d130e88ef9be0858b" exitCode=60
Oct 30 03:45:04.765527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.765347    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerDied","Data":"a1677e40cedba9258438e82d5b32161f53d3b94572873b7d130e88ef9be0858b"}
Oct 30 03:45:04.765684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.765659    4877 scope.go:122] "RemoveContainer" containerID="a1677e40cedba9258438e82d5b32161f53d3b94572873b7d130e88ef9be0858b"
Oct 30 03:45:04.768766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.768737    4877 generic.go:356] "Generic (PLEG): container finished" podID="99050aee-08a4-445e-8f13-1225366e18f0" containerID="e028e06ffd77c2db65247320d50070e35b53bd8916782a019ea7d42bb5b6da72" exitCode=60
Oct 30 03:45:04.768873 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.768831    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerDied","Data":"e028e06ffd77c2db65247320d50070e35b53bd8916782a019ea7d42bb5b6da72"}
Oct 30 03:45:04.769351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.769319    4877 scope.go:122] "RemoveContainer" containerID="e028e06ffd77c2db65247320d50070e35b53bd8916782a019ea7d42bb5b6da72"
Oct 30 03:45:04.785274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.785217    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" podStartSLOduration=4.785195391 podStartE2EDuration="4.785195391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:04.771066965 +0000 UTC m=+2533.199956383" watchObservedRunningTime="2026-10-30 03:45:04.785195391 +0000 UTC m=+2533.214084811"
Oct 30 03:45:04.844302 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.844250    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" podStartSLOduration=4.84423707 podStartE2EDuration="4.84423707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:04.842175847 +0000 UTC m=+2533.271065265" watchObservedRunningTime="2026-10-30 03:45:04.84423707 +0000 UTC m=+2533.273126487"
Oct 30 03:45:04.879938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.879878    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" podStartSLOduration=4.879863271 podStartE2EDuration="4.879863271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:04.877164472 +0000 UTC m=+2533.306054065" watchObservedRunningTime="2026-10-30 03:45:04.879863271 +0000 UTC m=+2533.308752712"
Oct 30 03:45:04.918636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:04.918588    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" podStartSLOduration=4.918573481 podStartE2EDuration="4.918573481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:04.914516764 +0000 UTC m=+2533.343406184" watchObservedRunningTime="2026-10-30 03:45:04.918573481 +0000 UTC m=+2533.347462934"
Oct 30 03:45:05.043056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.042995    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" podStartSLOduration=5.042980991 podStartE2EDuration="5.042980991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:05.041236461 +0000 UTC m=+2533.470125879" watchObservedRunningTime="2026-10-30 03:45:05.042980991 +0000 UTC m=+2533.471870408"
Oct 30 03:45:05.785197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.784953    4877 generic.go:356] "Generic (PLEG): container finished" podID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerID="5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc" exitCode=60
Oct 30 03:45:05.785197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.785047    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" event={"ID":"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739","Type":"ContainerDied","Data":"5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc"}
Oct 30 03:45:05.785197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.785098    4877 scope.go:122] "RemoveContainer" containerID="ce065c34dd652f422ddae7d6bc005b47687af849e65b9b70970e5e9bf0a02de1"
Oct 30 03:45:05.786059 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.785522    4877 scope.go:122] "RemoveContainer" containerID="5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc"
Oct 30 03:45:05.786059 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.785969    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp_odh-ai-gateway-infra(2a1b186d-1c21-4d11-8ea0-c17cdfd6b739)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739"
Oct 30 03:45:05.789041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.789015    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerStarted","Data":"20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618"}
Oct 30 03:45:05.791682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.791547    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerStarted","Data":"3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8"}
Oct 30 03:45:05.794398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.794370    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerStarted","Data":"beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5"}
Oct 30 03:45:05.798753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.798724    4877 generic.go:356] "Generic (PLEG): container finished" podID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerID="526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de" exitCode=60
Oct 30 03:45:05.799127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.799110    4877 scope.go:122] "RemoveContainer" containerID="526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de"
Oct 30 03:45:05.800664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.799644    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerDied","Data":"526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de"}
Oct 30 03:45:05.800664 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.800467    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j_odh-ai-gateway-infra(1732eae1-c0ac-4819-804d-b20403ff9d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e"
Oct 30 03:45:05.802787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.802763    4877 generic.go:356] "Generic (PLEG): container finished" podID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerID="9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4" exitCode=60
Oct 30 03:45:05.802892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.802842    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerDied","Data":"9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4"}
Oct 30 03:45:05.803129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.803109    4877 scope.go:122] "RemoveContainer" containerID="9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4"
Oct 30 03:45:05.803438 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.803400    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9_odh-ai-gateway-infra(b1178368-e2c2-4e73-bcb5-14e4a7106d0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f"
Oct 30 03:45:05.804341 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.804323    4877 scope.go:122] "RemoveContainer" containerID="4233c4b210820d2e2443beed60f3496f1293b66b7c861aa66088aca057a979d8"
Oct 30 03:45:05.805289 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.805191    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerStarted","Data":"f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a"}
Oct 30 03:45:05.807479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.807454    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerDied","Data":"7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a"}
Oct 30 03:45:05.807479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.807415    4877 generic.go:356] "Generic (PLEG): container finished" podID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerID="7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a" exitCode=60
Oct 30 03:45:05.807747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.807728    4877 scope.go:122] "RemoveContainer" containerID="7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a"
Oct 30 03:45:05.808025 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.808000    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8_odh-ai-gateway-infra(ebe8de4b-ae72-49e7-8cc3-d21faa913e02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02"
Oct 30 03:45:05.809948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.809918    4877 generic.go:356] "Generic (PLEG): container finished" podID="918ac658-590a-4014-aa4e-0cdb4a173213" containerID="d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a" exitCode=60
Oct 30 03:45:05.810036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.810006    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerDied","Data":"d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a"}
Oct 30 03:45:05.810256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.810236    4877 scope.go:122] "RemoveContainer" containerID="d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a"
Oct 30 03:45:05.810635 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.810598    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888865-57mfd_odh-ai-gateway-infra(918ac658-590a-4014-aa4e-0cdb4a173213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" podUID="918ac658-590a-4014-aa4e-0cdb4a173213"
Oct 30 03:45:05.812377 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.812353    4877 generic.go:356] "Generic (PLEG): container finished" podID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerID="68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9" exitCode=60
Oct 30 03:45:05.812521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.812494    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerDied","Data":"68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9"}
Oct 30 03:45:05.812665 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.812642    4877 scope.go:122] "RemoveContainer" containerID="68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9"
Oct 30 03:45:05.812915 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.812890    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846_odh-ai-gateway-infra(56795bbf-5ef2-463b-9bfa-1146e5b91ee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0"
Oct 30 03:45:05.814736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.814714    4877 generic.go:356] "Generic (PLEG): container finished" podID="a168bbc2-f646-40d7-897c-0531070a71a5" containerID="ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272" exitCode=60
Oct 30 03:45:05.814826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.814798    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerDied","Data":"ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272"}
Oct 30 03:45:05.815005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.814989    4877 scope.go:122] "RemoveContainer" containerID="ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272"
Oct 30 03:45:05.815252 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.815220    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn_odh-ai-gateway-infra(a168bbc2-f646-40d7-897c-0531070a71a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" podUID="a168bbc2-f646-40d7-897c-0531070a71a5"
Oct 30 03:45:05.817457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.817047    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerStarted","Data":"a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0"}
Oct 30 03:45:05.819287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.819260    4877 generic.go:356] "Generic (PLEG): container finished" podID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerID="3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1" exitCode=60
Oct 30 03:45:05.819373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.819292    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerDied","Data":"3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1"}
Oct 30 03:45:05.819644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.819628    4877 scope.go:122] "RemoveContainer" containerID="3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1"
Oct 30 03:45:05.819917 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:05.819896    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd_odh-ai-gateway-infra(a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb"
Oct 30 03:45:05.822904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.822207    4877 scope.go:122] "RemoveContainer" containerID="a2c692f079837f09237ac2af32cce439de7e5a625ae707200832e1449beb94e7"
Oct 30 03:45:05.838074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.838052    4877 scope.go:122] "RemoveContainer" containerID="9b22b00b104af2343e72b150dda10ebc2fb9fd0ca965f95a2e9106131ed78e8e"
Oct 30 03:45:05.853915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.853893    4877 scope.go:122] "RemoveContainer" containerID="2f6ca5ce71126686d3a86bdcf64d7ad3671f6d0e9e41a790befc0e7d99c1b157"
Oct 30 03:45:05.876206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.876103    4877 scope.go:122] "RemoveContainer" containerID="146b4e1b51615d86edfdb4be5f7e9e24a34952e03cbf0a3257a2753ffec2533f"
Oct 30 03:45:05.944811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.944783    4877 scope.go:122] "RemoveContainer" containerID="ec1a04768c652c87f7f630f7fe43475da3f678301bc75094c7568f2de6311b3f"
Oct 30 03:45:05.961926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:05.961904    4877 scope.go:122] "RemoveContainer" containerID="3472a4b83ba863e4c77a21a3ee8c36512c246898e89523d027f47e50b1a8c7ee"
Oct 30 03:45:06.838621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.838585    4877 scope.go:122] "RemoveContainer" containerID="5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc"
Oct 30 03:45:06.839097 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.839028    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp_odh-ai-gateway-infra(2a1b186d-1c21-4d11-8ea0-c17cdfd6b739)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739"
Oct 30 03:45:06.841134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.841101    4877 generic.go:356] "Generic (PLEG): container finished" podID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerID="20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618" exitCode=60
Oct 30 03:45:06.841273 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.841180    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerDied","Data":"20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618"}
Oct 30 03:45:06.841340 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.841276    4877 scope.go:122] "RemoveContainer" containerID="a1677e40cedba9258438e82d5b32161f53d3b94572873b7d130e88ef9be0858b"
Oct 30 03:45:06.841494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.841471    4877 scope.go:122] "RemoveContainer" containerID="20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618"
Oct 30 03:45:06.841819 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.841789    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7_odh-ai-gateway-infra(6cecb738-c2c3-442e-86c0-a1a6745ca7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff"
Oct 30 03:45:06.843537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.843514    4877 generic.go:356] "Generic (PLEG): container finished" podID="99050aee-08a4-445e-8f13-1225366e18f0" containerID="3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8" exitCode=60
Oct 30 03:45:06.843632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.843599    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerDied","Data":"3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8"}
Oct 30 03:45:06.843807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.843794    4877 scope.go:122] "RemoveContainer" containerID="3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8"
Oct 30 03:45:06.844053 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.844018    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf_odh-ai-gateway-infra(99050aee-08a4-445e-8f13-1225366e18f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" podUID="99050aee-08a4-445e-8f13-1225366e18f0"
Oct 30 03:45:06.845596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.845573    4877 generic.go:356] "Generic (PLEG): container finished" podID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerID="beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5" exitCode=60
Oct 30 03:45:06.845682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.845670    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerDied","Data":"beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5"}
Oct 30 03:45:06.845847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.845831    4877 scope.go:122] "RemoveContainer" containerID="beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5"
Oct 30 03:45:06.846079 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.846061    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc_odh-ai-gateway-infra(09029590-3cdc-4fa7-b239-79c5db99fa0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e"
Oct 30 03:45:06.847696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.847676    4877 scope.go:122] "RemoveContainer" containerID="526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de"
Oct 30 03:45:06.847957 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.847933    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j_odh-ai-gateway-infra(1732eae1-c0ac-4819-804d-b20403ff9d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e"
Oct 30 03:45:06.849258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.849241    4877 scope.go:122] "RemoveContainer" containerID="9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4"
Oct 30 03:45:06.849510 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.849479    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9_odh-ai-gateway-infra(b1178368-e2c2-4e73-bcb5-14e4a7106d0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f"
Oct 30 03:45:06.851023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.851003    4877 generic.go:356] "Generic (PLEG): container finished" podID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerID="f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a" exitCode=60
Oct 30 03:45:06.851114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.851076    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerDied","Data":"f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a"}
Oct 30 03:45:06.851310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.851296    4877 scope.go:122] "RemoveContainer" containerID="f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a"
Oct 30 03:45:06.851545 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.851522    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq_odh-ai-gateway-infra(f0d2cc44-4d44-4800-af7c-fe1e252b4b3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a"
Oct 30 03:45:06.853400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.853375    4877 generic.go:356] "Generic (PLEG): container finished" podID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerID="896199a68f3baab8372a8313fbde64c81a93868a6dad2e595061a67474a8ec93" exitCode=60
Oct 30 03:45:06.853547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.853465    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" event={"ID":"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8","Type":"ContainerDied","Data":"896199a68f3baab8372a8313fbde64c81a93868a6dad2e595061a67474a8ec93"}
Oct 30 03:45:06.853662 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.853646    4877 scope.go:122] "RemoveContainer" containerID="896199a68f3baab8372a8313fbde64c81a93868a6dad2e595061a67474a8ec93"
Oct 30 03:45:06.853901 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.853874    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h_odh-ai-gateway-infra(7b0e78d1-c76f-4108-88a3-2fb9158e2bf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8"
Oct 30 03:45:06.855314 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.855294    4877 scope.go:122] "RemoveContainer" containerID="7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a"
Oct 30 03:45:06.855498 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.855476    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8_odh-ai-gateway-infra(ebe8de4b-ae72-49e7-8cc3-d21faa913e02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02"
Oct 30 03:45:06.856951 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.856933    4877 scope.go:122] "RemoveContainer" containerID="d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a"
Oct 30 03:45:06.857199 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.857179    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888865-57mfd_odh-ai-gateway-infra(918ac658-590a-4014-aa4e-0cdb4a173213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" podUID="918ac658-590a-4014-aa4e-0cdb4a173213"
Oct 30 03:45:06.857871 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.857858    4877 scope.go:122] "RemoveContainer" containerID="e028e06ffd77c2db65247320d50070e35b53bd8916782a019ea7d42bb5b6da72"
Oct 30 03:45:06.858879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.858862    4877 generic.go:356] "Generic (PLEG): container finished" podID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerID="9b0f6b15db355fddf5bf852c7fc9bea29ae2da5d9ef68be3b67cee37ae781572" exitCode=60
Oct 30 03:45:06.858958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.858935    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerDied","Data":"9b0f6b15db355fddf5bf852c7fc9bea29ae2da5d9ef68be3b67cee37ae781572"}
Oct 30 03:45:06.859141 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.859124    4877 scope.go:122] "RemoveContainer" containerID="9b0f6b15db355fddf5bf852c7fc9bea29ae2da5d9ef68be3b67cee37ae781572"
Oct 30 03:45:06.859349 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.859329    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj_odh-ai-gateway-infra(38de7398-445d-4e13-9934-6fc61ed7b16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b"
Oct 30 03:45:06.860739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.860722    4877 scope.go:122] "RemoveContainer" containerID="68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9"
Oct 30 03:45:06.860945 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.860926    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846_odh-ai-gateway-infra(56795bbf-5ef2-463b-9bfa-1146e5b91ee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0"
Oct 30 03:45:06.862330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.862314    4877 scope.go:122] "RemoveContainer" containerID="ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272"
Oct 30 03:45:06.862553 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.862533    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn_odh-ai-gateway-infra(a168bbc2-f646-40d7-897c-0531070a71a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" podUID="a168bbc2-f646-40d7-897c-0531070a71a5"
Oct 30 03:45:06.864058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.864032    4877 generic.go:356] "Generic (PLEG): container finished" podID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerID="a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0" exitCode=60
Oct 30 03:45:06.864178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.864113    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerDied","Data":"a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0"}
Oct 30 03:45:06.864300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.864281    4877 scope.go:122] "RemoveContainer" containerID="a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0"
Oct 30 03:45:06.864548 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.864527    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g_odh-ai-gateway-infra(97413284-b6ba-44f3-ab43-66f0ff82268b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b"
Oct 30 03:45:06.866024 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.866007    4877 scope.go:122] "RemoveContainer" containerID="3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1"
Oct 30 03:45:06.866208 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:06.866191    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd_odh-ai-gateway-infra(a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb"
Oct 30 03:45:06.873139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.873120    4877 scope.go:122] "RemoveContainer" containerID="0fd4b076b99a3a1f0918ba2d331f71235ab36d48cef9707075524642f197d637"
Oct 30 03:45:06.888214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.888193    4877 scope.go:122] "RemoveContainer" containerID="cd63df22fcb5cba409a561771b5b8af25559595f197ad4a0fae7bab33412d9a7"
Oct 30 03:45:06.903551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.903522    4877 scope.go:122] "RemoveContainer" containerID="e7d46a007246d2fb4b3bb40b3b80327b7e30af4d33f296ca4409c4faf772f79c"
Oct 30 03:45:06.919180 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.919155    4877 scope.go:122] "RemoveContainer" containerID="a8827c0460442bf574d6c16ef36dd8ff746aa3d2f5e1cbae2e0abef6ff51c8d5"
Oct 30 03:45:06.934940 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:06.934909    4877 scope.go:122] "RemoveContainer" containerID="d1f8ad2d99bc9ab391b1e2ccbeab94597b38c988af642a858243072b5c2791ca"
Oct 30 03:45:07.878399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.878362    4877 scope.go:122] "RemoveContainer" containerID="20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618"
Oct 30 03:45:07.878863 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.878815    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7_odh-ai-gateway-infra(6cecb738-c2c3-442e-86c0-a1a6745ca7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff"
Oct 30 03:45:07.880171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.880147    4877 scope.go:122] "RemoveContainer" containerID="3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8"
Oct 30 03:45:07.880537 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.880514    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf_odh-ai-gateway-infra(99050aee-08a4-445e-8f13-1225366e18f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" podUID="99050aee-08a4-445e-8f13-1225366e18f0"
Oct 30 03:45:07.881906 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.881886    4877 scope.go:122] "RemoveContainer" containerID="beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5"
Oct 30 03:45:07.882193 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.882166    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc_odh-ai-gateway-infra(09029590-3cdc-4fa7-b239-79c5db99fa0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e"
Oct 30 03:45:07.883609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.883586    4877 scope.go:122] "RemoveContainer" containerID="f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a"
Oct 30 03:45:07.883865 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.883841    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq_odh-ai-gateway-infra(f0d2cc44-4d44-4800-af7c-fe1e252b4b3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a"
Oct 30 03:45:07.885334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.885316    4877 scope.go:122] "RemoveContainer" containerID="896199a68f3baab8372a8313fbde64c81a93868a6dad2e595061a67474a8ec93"
Oct 30 03:45:07.885609 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.885580    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h_odh-ai-gateway-infra(7b0e78d1-c76f-4108-88a3-2fb9158e2bf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8"
Oct 30 03:45:07.887153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.887131    4877 scope.go:122] "RemoveContainer" containerID="9b0f6b15db355fddf5bf852c7fc9bea29ae2da5d9ef68be3b67cee37ae781572"
Oct 30 03:45:07.887419 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.887393    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj_odh-ai-gateway-infra(38de7398-445d-4e13-9934-6fc61ed7b16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b"
Oct 30 03:45:07.889193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:07.889173    4877 scope.go:122] "RemoveContainer" containerID="a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0"
Oct 30 03:45:07.889408 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:07.889388    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g_odh-ai-gateway-infra(97413284-b6ba-44f3-ab43-66f0ff82268b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b"
Oct 30 03:45:10.030864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:10.030818    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"]
Oct 30 03:45:10.035658 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:10.035631    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45320976-5br2f"]
Oct 30 03:45:11.030726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:11.030691    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"]
Oct 30 03:45:11.042728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:11.042691    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c58e48da-79r44"]
Oct 30 03:45:11.702903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:11.702868    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ccca04-f4e0-4c3d-aaa8-213934cec323" path="/var/lib/kubelet/pods/c0ccca04-f4e0-4c3d-aaa8-213934cec323/volumes"
Oct 30 03:45:11.703363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:11.703347    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5009972-a6ad-4988-909f-260e3c8b1c6c" path="/var/lib/kubelet/pods/e5009972-a6ad-4988-909f-260e3c8b1c6c/volumes"
Oct 30 03:45:16.250208 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:16.250163    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:45:16.690991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:16.690948    4877 scope.go:122] "RemoveContainer" containerID="20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618"
Oct 30 03:45:16.691371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:16.691347    4877 scope.go:122] "RemoveContainer" containerID="ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272"
Oct 30 03:45:16.691596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:16.691574    4877 scope.go:122] "RemoveContainer" containerID="5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc"
Oct 30 03:45:17.695297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:17.695263    4877 scope.go:122] "RemoveContainer" containerID="7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a"
Oct 30 03:45:17.696748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:17.696227    4877 scope.go:122] "RemoveContainer" containerID="3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1"
Oct 30 03:45:17.696951 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:17.696930    4877 scope.go:122] "RemoveContainer" containerID="d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a"
Oct 30 03:45:17.698074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:17.697663    4877 scope.go:122] "RemoveContainer" containerID="9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4"
Oct 30 03:45:17.700810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:17.700787    4877 scope.go:122] "RemoveContainer" containerID="3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8"
Oct 30 03:45:17.701400 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:17.701375    4877 scope.go:122] "RemoveContainer" containerID="526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de"
Oct 30 03:45:18.086944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.086893    4877 generic.go:356] "Generic (PLEG): container finished" podID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerID="7e89594a0aa02893bd5f638ede3b6e11670e990862f1fdf08ff34c6c68bab932" exitCode=60
Oct 30 03:45:18.087122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.086966    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" event={"ID":"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739","Type":"ContainerDied","Data":"7e89594a0aa02893bd5f638ede3b6e11670e990862f1fdf08ff34c6c68bab932"}
Oct 30 03:45:18.087122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.087024    4877 scope.go:122] "RemoveContainer" containerID="5836b676ce1024ae54bd69645b4b04d0eb6955ad4116c055edb87721aa3bdabc"
Oct 30 03:45:18.087551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.087531    4877 scope.go:122] "RemoveContainer" containerID="7e89594a0aa02893bd5f638ede3b6e11670e990862f1fdf08ff34c6c68bab932"
Oct 30 03:45:18.087909 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:18.087887    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp_odh-ai-gateway-infra(2a1b186d-1c21-4d11-8ea0-c17cdfd6b739)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739"
Oct 30 03:45:18.091393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.091347    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerStarted","Data":"514016b4e28912addcfa6758f1b052651ccc70d6488f988f2b829fda7aca3df6"}
Oct 30 03:45:18.097653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.097626    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerStarted","Data":"8bbdc039322bb8ff9d748ea98fa58c2a831edcf10d69a1a5f79618f363cc7528"}
Oct 30 03:45:18.691173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.691138    4877 scope.go:122] "RemoveContainer" containerID="f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a"
Oct 30 03:45:18.691552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.691529    4877 scope.go:122] "RemoveContainer" containerID="68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9"
Oct 30 03:45:18.691904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:18.691878    4877 scope.go:122] "RemoveContainer" containerID="896199a68f3baab8372a8313fbde64c81a93868a6dad2e595061a67474a8ec93"
Oct 30 03:45:19.110250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.110217    4877 generic.go:356] "Generic (PLEG): container finished" podID="918ac658-590a-4014-aa4e-0cdb4a173213" containerID="e8344bf5a11141c63568ab5adaf5970467ce584e5bb23528b70c55f8474fbfcf" exitCode=60
Oct 30 03:45:19.110742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.110256    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerDied","Data":"e8344bf5a11141c63568ab5adaf5970467ce584e5bb23528b70c55f8474fbfcf"}
Oct 30 03:45:19.110742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.110321    4877 scope.go:122] "RemoveContainer" containerID="d17599c68eaf1d5d7504aca87492fd8105556ffaa6cc4e6238eae4bf9b00c91a"
Oct 30 03:45:19.110934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.110913    4877 scope.go:122] "RemoveContainer" containerID="e8344bf5a11141c63568ab5adaf5970467ce584e5bb23528b70c55f8474fbfcf"
Oct 30 03:45:19.111366 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:19.111338    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888865-57mfd_odh-ai-gateway-infra(918ac658-590a-4014-aa4e-0cdb4a173213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" podUID="918ac658-590a-4014-aa4e-0cdb4a173213"
Oct 30 03:45:19.113742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.113718    4877 generic.go:356] "Generic (PLEG): container finished" podID="a168bbc2-f646-40d7-897c-0531070a71a5" containerID="8bbdc039322bb8ff9d748ea98fa58c2a831edcf10d69a1a5f79618f363cc7528" exitCode=60
Oct 30 03:45:19.113856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.113790    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerDied","Data":"8bbdc039322bb8ff9d748ea98fa58c2a831edcf10d69a1a5f79618f363cc7528"}
Oct 30 03:45:19.114208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.114186    4877 scope.go:122] "RemoveContainer" containerID="8bbdc039322bb8ff9d748ea98fa58c2a831edcf10d69a1a5f79618f363cc7528"
Oct 30 03:45:19.114602 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:19.114580    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn_odh-ai-gateway-infra(a168bbc2-f646-40d7-897c-0531070a71a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" podUID="a168bbc2-f646-40d7-897c-0531070a71a5"
Oct 30 03:45:19.116185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.116152    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerStarted","Data":"043c49a4c0d2fc9eb54851a15f5febd6a4d32fcec2afb1b35a07e73fc64f9efc"}
Oct 30 03:45:19.120211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.120189    4877 generic.go:356] "Generic (PLEG): container finished" podID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerID="514016b4e28912addcfa6758f1b052651ccc70d6488f988f2b829fda7aca3df6" exitCode=60
Oct 30 03:45:19.120295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.120256    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerDied","Data":"514016b4e28912addcfa6758f1b052651ccc70d6488f988f2b829fda7aca3df6"}
Oct 30 03:45:19.120677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.120654    4877 scope.go:122] "RemoveContainer" containerID="514016b4e28912addcfa6758f1b052651ccc70d6488f988f2b829fda7aca3df6"
Oct 30 03:45:19.120988 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:19.120966    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7_odh-ai-gateway-infra(6cecb738-c2c3-442e-86c0-a1a6745ca7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff"
Oct 30 03:45:19.122345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.122323    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerStarted","Data":"d5e629b05608afca3c98b518261714f9db2e2343b96d26b4586a62f9b6156eb7"}
Oct 30 03:45:19.124341 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.124313    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerStarted","Data":"c66104bff627e764a09ffb6bac94c40aa0a1c31da7e3ae9a98effd51c1c85034"}
Oct 30 03:45:19.126389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.126367    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerStarted","Data":"c253ddcea4b08cfe42d1ec4b07f3dcf096aa600edd7370c5d56f698931d50c85"}
Oct 30 03:45:19.128503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.128481    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerStarted","Data":"1cec73926b3d58ac11c62462dffdb70900f2fe5d3c78ccb21b84b44bede8577c"}
Oct 30 03:45:19.141291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.141261    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"]
Oct 30 03:45:19.172071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.172030    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"]
Oct 30 03:45:19.212389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.212357    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"]
Oct 30 03:45:19.528110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.528062    4877 scope.go:122] "RemoveContainer" containerID="ce0a2ea9bdf54b8fdfaade5ce69278838ae2e3aded6daf18f193e85044255272"
Oct 30 03:45:19.536110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.536077    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:19.550287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.550172    4877 scope.go:122] "RemoveContainer" containerID="20abb5fd22df999908fbb2e483324ad8c4ed7f34b7d86988953ae5ab60e54618"
Oct 30 03:45:19.684366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.684332    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc\" (UniqueName: \"kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc\") pod \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") "
Oct 30 03:45:19.684572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.684391    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca\") pod \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\" (UID: \"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739\") "
Oct 30 03:45:19.684810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.684775    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca" pod "2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" (UID: "2a1b186d-1c21-4d11-8ea0-c17cdfd6b739"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:19.685551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.685523    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:19.686922 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.686889    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc" pod "2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" (UID: "2a1b186d-1c21-4d11-8ea0-c17cdfd6b739"). InnerVolumeSpecName "kube-api-access-2lsrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:19.691695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.691669    4877 scope.go:122] "RemoveContainer" containerID="a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0"
Oct 30 03:45:19.692274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.692254    4877 scope.go:122] "RemoveContainer" containerID="9b0f6b15db355fddf5bf852c7fc9bea29ae2da5d9ef68be3b67cee37ae781572"
Oct 30 03:45:19.787285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:19.787241    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lsrc\" (UniqueName: \"kubernetes.io/projected/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739-kube-api-access-2lsrc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.147697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.147656    4877 generic.go:356] "Generic (PLEG): container finished" podID="99050aee-08a4-445e-8f13-1225366e18f0" containerID="d5e629b05608afca3c98b518261714f9db2e2343b96d26b4586a62f9b6156eb7" exitCode=60
Oct 30 03:45:20.148176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.147807    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerDied","Data":"d5e629b05608afca3c98b518261714f9db2e2343b96d26b4586a62f9b6156eb7"}
Oct 30 03:45:20.148176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.147871    4877 scope.go:122] "RemoveContainer" containerID="3212e498965c598a223adc798a0738e1d5a1c3afad370a8cf2839f2f44175bf8"
Oct 30 03:45:20.148396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.148373    4877 scope.go:122] "RemoveContainer" containerID="d5e629b05608afca3c98b518261714f9db2e2343b96d26b4586a62f9b6156eb7"
Oct 30 03:45:20.149634 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.148845    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf_odh-ai-gateway-infra(99050aee-08a4-445e-8f13-1225366e18f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" podUID="99050aee-08a4-445e-8f13-1225366e18f0"
Oct 30 03:45:20.151776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.151751    4877 generic.go:356] "Generic (PLEG): container finished" podID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerID="c66104bff627e764a09ffb6bac94c40aa0a1c31da7e3ae9a98effd51c1c85034" exitCode=60
Oct 30 03:45:20.151865 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.151826    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerDied","Data":"c66104bff627e764a09ffb6bac94c40aa0a1c31da7e3ae9a98effd51c1c85034"}
Oct 30 03:45:20.152267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.152246    4877 scope.go:122] "RemoveContainer" containerID="c66104bff627e764a09ffb6bac94c40aa0a1c31da7e3ae9a98effd51c1c85034"
Oct 30 03:45:20.152658 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.152622    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j_odh-ai-gateway-infra(1732eae1-c0ac-4819-804d-b20403ff9d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e"
Oct 30 03:45:20.154754 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.154719    4877 generic.go:356] "Generic (PLEG): container finished" podID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerID="c253ddcea4b08cfe42d1ec4b07f3dcf096aa600edd7370c5d56f698931d50c85" exitCode=60
Oct 30 03:45:20.154849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.154800    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerDied","Data":"c253ddcea4b08cfe42d1ec4b07f3dcf096aa600edd7370c5d56f698931d50c85"}
Oct 30 03:45:20.155298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.155275    4877 scope.go:122] "RemoveContainer" containerID="c253ddcea4b08cfe42d1ec4b07f3dcf096aa600edd7370c5d56f698931d50c85"
Oct 30 03:45:20.155730 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.155700    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9_odh-ai-gateway-infra(b1178368-e2c2-4e73-bcb5-14e4a7106d0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f"
Oct 30 03:45:20.157214 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.157195    4877 generic.go:356] "Generic (PLEG): container finished" podID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerID="08eb07ea515a824786fb84f846fbede9eb0a8b7364b8a4704087003a5c39dbd6" exitCode=60
Oct 30 03:45:20.157295 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.157265    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerDied","Data":"08eb07ea515a824786fb84f846fbede9eb0a8b7364b8a4704087003a5c39dbd6"}
Oct 30 03:45:20.157635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.157618    4877 scope.go:122] "RemoveContainer" containerID="08eb07ea515a824786fb84f846fbede9eb0a8b7364b8a4704087003a5c39dbd6"
Oct 30 03:45:20.157978 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.157954    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq_odh-ai-gateway-infra(f0d2cc44-4d44-4800-af7c-fe1e252b4b3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a"
Oct 30 03:45:20.159623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.159601    4877 generic.go:356] "Generic (PLEG): container finished" podID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerID="21bfb23d9ba2a9c89d1309b3212203bad64d104544aa49af773d5743bde9e73e" exitCode=60
Oct 30 03:45:20.159731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.159700    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" event={"ID":"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8","Type":"ContainerDied","Data":"21bfb23d9ba2a9c89d1309b3212203bad64d104544aa49af773d5743bde9e73e"}
Oct 30 03:45:20.160066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.160025    4877 scope.go:122] "RemoveContainer" containerID="21bfb23d9ba2a9c89d1309b3212203bad64d104544aa49af773d5743bde9e73e"
Oct 30 03:45:20.160712 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.160290    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h_odh-ai-gateway-infra(7b0e78d1-c76f-4108-88a3-2fb9158e2bf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8"
Oct 30 03:45:20.161072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.161044    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"]
Oct 30 03:45:20.162601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.162580    4877 generic.go:356] "Generic (PLEG): container finished" podID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerID="1cec73926b3d58ac11c62462dffdb70900f2fe5d3c78ccb21b84b44bede8577c" exitCode=60
Oct 30 03:45:20.162695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.162646    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerDied","Data":"1cec73926b3d58ac11c62462dffdb70900f2fe5d3c78ccb21b84b44bede8577c"}
Oct 30 03:45:20.163003 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.162984    4877 scope.go:122] "RemoveContainer" containerID="1cec73926b3d58ac11c62462dffdb70900f2fe5d3c78ccb21b84b44bede8577c"
Oct 30 03:45:20.163220 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.163197    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8_odh-ai-gateway-infra(ebe8de4b-ae72-49e7-8cc3-d21faa913e02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02"
Oct 30 03:45:20.167017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.166995    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerStarted","Data":"0e000a78527766c104b0782f0720d4c51af61e8533b239814e35fe55b7cd10de"}
Oct 30 03:45:20.168867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.168849    4877 scope.go:122] "RemoveContainer" containerID="526dd1ff855cfb0d11309ad3fc0b50cfe4dc202f61322a20a0ee96c4cbec54de"
Oct 30 03:45:20.169601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.169577    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerStarted","Data":"c5130cbacf5f37dafa6656adf8abaeb266c71bedddb70e0aecf76c8fd5dab153"}
Oct 30 03:45:20.173776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.173757    4877 generic.go:356] "Generic (PLEG): container finished" podID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerID="043c49a4c0d2fc9eb54851a15f5febd6a4d32fcec2afb1b35a07e73fc64f9efc" exitCode=60
Oct 30 03:45:20.173849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.173823    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerDied","Data":"043c49a4c0d2fc9eb54851a15f5febd6a4d32fcec2afb1b35a07e73fc64f9efc"}
Oct 30 03:45:20.174144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.174126    4877 scope.go:122] "RemoveContainer" containerID="043c49a4c0d2fc9eb54851a15f5febd6a4d32fcec2afb1b35a07e73fc64f9efc"
Oct 30 03:45:20.174351 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:20.174331    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd_odh-ai-gateway-infra(a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb"
Oct 30 03:45:20.175574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.175553    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp" event={"ID":"2a1b186d-1c21-4d11-8ea0-c17cdfd6b739","Type":"ContainerDied","Data":"9adb0ea5440e5e6b4aa8eff7d75bd7818e0e3f60c41f79aa43bdb7566c590695"}
Oct 30 03:45:20.175790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.175624    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"
Oct 30 03:45:20.197388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.197361    4877 scope.go:122] "RemoveContainer" containerID="9dcf76890dcdeb1b96e642fdecfd6241f702a128d4016c39d08f1173588078a4"
Oct 30 03:45:20.202551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.202516    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"]
Oct 30 03:45:20.215579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.215559    4877 scope.go:122] "RemoveContainer" containerID="f04f65a70c9b3b564feda07ee7d6a4a4033f69f07bca7e90afbee28fb2c1fe6a"
Oct 30 03:45:20.231802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.231776    4877 scope.go:122] "RemoveContainer" containerID="896199a68f3baab8372a8313fbde64c81a93868a6dad2e595061a67474a8ec93"
Oct 30 03:45:20.250467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.250406    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"]
Oct 30 03:45:20.255476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.255416    4877 scope.go:122] "RemoveContainer" containerID="7696f8c1f4e122ad07f58d329d7541e8ed72e481d389f86c89bb03c373e8247a"
Oct 30 03:45:20.279320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.279253    4877 scope.go:122] "RemoveContainer" containerID="3f86e4388bf2dd8bf103ef79ba75403850731466f6da6c87bbce853437f0a3b1"
Oct 30 03:45:20.315073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.315013    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"]
Oct 30 03:45:20.323443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.323399    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7d99a4-29888865-4ddsp"]
Oct 30 03:45:20.672525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.670704    4877 scope.go:122] "RemoveContainer" containerID="7e89594a0aa02893bd5f638ede3b6e11670e990862f1fdf08ff34c6c68bab932"
Oct 30 03:45:20.703460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:20.697121    4877 scope.go:122] "RemoveContainer" containerID="beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5"
Oct 30 03:45:21.210348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.210266    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:21.229248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.229207    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca\") pod \"918ac658-590a-4014-aa4e-0cdb4a173213\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") "
Oct 30 03:45:21.229462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.229292    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg\" (UniqueName: \"kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg\") pod \"918ac658-590a-4014-aa4e-0cdb4a173213\" (UID: \"918ac658-590a-4014-aa4e-0cdb4a173213\") "
Oct 30 03:45:21.232604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.232560    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca" pod "918ac658-590a-4014-aa4e-0cdb4a173213" (UID: "918ac658-590a-4014-aa4e-0cdb4a173213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.233879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.233843    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg" pod "918ac658-590a-4014-aa4e-0cdb4a173213" (UID: "918ac658-590a-4014-aa4e-0cdb4a173213"). InnerVolumeSpecName "kube-api-access-9w6wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.241196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.241106    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"]
Oct 30 03:45:21.248373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.248347    4877 generic.go:356] "Generic (PLEG): container finished" podID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerID="0e000a78527766c104b0782f0720d4c51af61e8533b239814e35fe55b7cd10de" exitCode=60
Oct 30 03:45:21.248500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.248416    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerDied","Data":"0e000a78527766c104b0782f0720d4c51af61e8533b239814e35fe55b7cd10de"}
Oct 30 03:45:21.248565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.248506    4877 scope.go:122] "RemoveContainer" containerID="9b0f6b15db355fddf5bf852c7fc9bea29ae2da5d9ef68be3b67cee37ae781572"
Oct 30 03:45:21.248981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.248960    4877 scope.go:122] "RemoveContainer" containerID="0e000a78527766c104b0782f0720d4c51af61e8533b239814e35fe55b7cd10de"
Oct 30 03:45:21.249390 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:21.249364    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj_odh-ai-gateway-infra(38de7398-445d-4e13-9934-6fc61ed7b16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b"
Oct 30 03:45:21.251503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.251415    4877 generic.go:356] "Generic (PLEG): container finished" podID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerID="c5130cbacf5f37dafa6656adf8abaeb266c71bedddb70e0aecf76c8fd5dab153" exitCode=60
Oct 30 03:45:21.251560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.251500    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerDied","Data":"c5130cbacf5f37dafa6656adf8abaeb266c71bedddb70e0aecf76c8fd5dab153"}
Oct 30 03:45:21.251955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.251933    4877 scope.go:122] "RemoveContainer" containerID="c5130cbacf5f37dafa6656adf8abaeb266c71bedddb70e0aecf76c8fd5dab153"
Oct 30 03:45:21.252312 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:21.252287    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846_odh-ai-gateway-infra(56795bbf-5ef2-463b-9bfa-1146e5b91ee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0"
Oct 30 03:45:21.253900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.253876    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerStarted","Data":"0e6b35e1a6ac762206350668d348bc4d80e7ed2bd2bd2d4d78c412a312c3b1f6"}
Oct 30 03:45:21.254465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.254448    4877 scope.go:122] "RemoveContainer" containerID="0e6b35e1a6ac762206350668d348bc4d80e7ed2bd2bd2d4d78c412a312c3b1f6"
Oct 30 03:45:21.254914 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:21.254887    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g_odh-ai-gateway-infra(97413284-b6ba-44f3-ab43-66f0ff82268b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b"
Oct 30 03:45:21.257897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.257876    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"]
Oct 30 03:45:21.280311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.280287    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"]
Oct 30 03:45:21.282000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.281978    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd" event={"ID":"918ac658-590a-4014-aa4e-0cdb4a173213","Type":"ContainerDied","Data":"3447c7e03aea979140e4fdd6f5242f07963fca5776d90e14d6db9564947803b2"}
Oct 30 03:45:21.282185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.282170    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"
Oct 30 03:45:21.328708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.328674    4877 scope.go:122] "RemoveContainer" containerID="68b9495aae2135e36416b0defb7fbcca1d475130b87be9a163a3b48a17b5bee9"
Oct 30 03:45:21.333349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.333310    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918ac658-590a-4014-aa4e-0cdb4a173213-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.333607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.333589    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6wg\" (UniqueName: \"kubernetes.io/projected/918ac658-590a-4014-aa4e-0cdb4a173213-kube-api-access-9w6wg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.445938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.445573    4877 scope.go:122] "RemoveContainer" containerID="e8344bf5a11141c63568ab5adaf5970467ce584e5bb23528b70c55f8474fbfcf"
Oct 30 03:45:21.478253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.474366    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"]
Oct 30 03:45:21.507637 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.499648    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:21.507637 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.505275    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:21.531365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.531127    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"]
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.540080    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f\" (UniqueName: \"kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f\") pod \"a168bbc2-f646-40d7-897c-0531070a71a5\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") "
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.540244    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca\") pod \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") "
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.540392    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888865-57mfd"]
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.540496    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca\") pod \"a168bbc2-f646-40d7-897c-0531070a71a5\" (UID: \"a168bbc2-f646-40d7-897c-0531070a71a5\") "
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.540603    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg\" (UniqueName: \"kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg\") pod \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\" (UID: \"6cecb738-c2c3-442e-86c0-a1a6745ca7ff\") "
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.541161    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca" pod "6cecb738-c2c3-442e-86c0-a1a6745ca7ff" (UID: "6cecb738-c2c3-442e-86c0-a1a6745ca7ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.541520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.541381    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.542241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.541793    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca" pod "a168bbc2-f646-40d7-897c-0531070a71a5" (UID: "a168bbc2-f646-40d7-897c-0531070a71a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.543970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.543768    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg" pod "6cecb738-c2c3-442e-86c0-a1a6745ca7ff" (UID: "6cecb738-c2c3-442e-86c0-a1a6745ca7ff"). InnerVolumeSpecName "kube-api-access-gnqxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.547978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.547961    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f" pod "a168bbc2-f646-40d7-897c-0531070a71a5" (UID: "a168bbc2-f646-40d7-897c-0531070a71a5"). InnerVolumeSpecName "kube-api-access-5vk6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.626647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.626618    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:21.642213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.642175    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h\" (UniqueName: \"kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h\") pod \"1732eae1-c0ac-4819-804d-b20403ff9d6e\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") "
Oct 30 03:45:21.642389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.642353    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca\") pod \"1732eae1-c0ac-4819-804d-b20403ff9d6e\" (UID: \"1732eae1-c0ac-4819-804d-b20403ff9d6e\") "
Oct 30 03:45:21.643234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.643212    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vk6f\" (UniqueName: \"kubernetes.io/projected/a168bbc2-f646-40d7-897c-0531070a71a5-kube-api-access-5vk6f\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.643360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.643235    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a168bbc2-f646-40d7-897c-0531070a71a5-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.643360 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.643250    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnqxg\" (UniqueName: \"kubernetes.io/projected/6cecb738-c2c3-442e-86c0-a1a6745ca7ff-kube-api-access-gnqxg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.644578 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.643690    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca" pod "1732eae1-c0ac-4819-804d-b20403ff9d6e" (UID: "1732eae1-c0ac-4819-804d-b20403ff9d6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.708877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.708834    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h" pod "1732eae1-c0ac-4819-804d-b20403ff9d6e" (UID: "1732eae1-c0ac-4819-804d-b20403ff9d6e"). InnerVolumeSpecName "kube-api-access-6dj9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.721583 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.721547    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" path="/var/lib/kubelet/pods/2a1b186d-1c21-4d11-8ea0-c17cdfd6b739/volumes"
Oct 30 03:45:21.722831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.722323    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" path="/var/lib/kubelet/pods/918ac658-590a-4014-aa4e-0cdb4a173213/volumes"
Oct 30 03:45:21.739201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.739180    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:21.745468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.744486    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch\" (UniqueName: \"kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch\") pod \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") "
Oct 30 03:45:21.745468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.744743    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca\") pod \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\" (UID: \"ebe8de4b-ae72-49e7-8cc3-d21faa913e02\") "
Oct 30 03:45:21.745468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.745368    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dj9h\" (UniqueName: \"kubernetes.io/projected/1732eae1-c0ac-4819-804d-b20403ff9d6e-kube-api-access-6dj9h\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.745468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.745391    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1732eae1-c0ac-4819-804d-b20403ff9d6e-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.746112 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.746090    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca" pod "ebe8de4b-ae72-49e7-8cc3-d21faa913e02" (UID: "ebe8de4b-ae72-49e7-8cc3-d21faa913e02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.748739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.748698    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch" pod "ebe8de4b-ae72-49e7-8cc3-d21faa913e02" (UID: "ebe8de4b-ae72-49e7-8cc3-d21faa913e02"). InnerVolumeSpecName "kube-api-access-447ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.752371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.752351    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:21.776013 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.774601    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"]
Oct 30 03:45:21.845950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.845920    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9\" (UniqueName: \"kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9\") pod \"99050aee-08a4-445e-8f13-1225366e18f0\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") "
Oct 30 03:45:21.846128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.846007    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca\") pod \"99050aee-08a4-445e-8f13-1225366e18f0\" (UID: \"99050aee-08a4-445e-8f13-1225366e18f0\") "
Oct 30 03:45:21.846802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.846774    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.846912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.846807    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447ch\" (UniqueName: \"kubernetes.io/projected/ebe8de4b-ae72-49e7-8cc3-d21faa913e02-kube-api-access-447ch\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.846912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.846881    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca" pod "99050aee-08a4-445e-8f13-1225366e18f0" (UID: "99050aee-08a4-445e-8f13-1225366e18f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.848263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.848238    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9" pod "99050aee-08a4-445e-8f13-1225366e18f0" (UID: "99050aee-08a4-445e-8f13-1225366e18f0"). InnerVolumeSpecName "kube-api-access-sp2h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.894015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.893986    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:21.898699 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.898672    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:21.920038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.904824    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"]
Oct 30 03:45:21.948244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.948201    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca\") pod \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") "
Oct 30 03:45:21.948471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.948413    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk\" (UniqueName: \"kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk\") pod \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") "
Oct 30 03:45:21.948585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.948503    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8\" (UniqueName: \"kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8\") pod \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\" (UID: \"b1178368-e2c2-4e73-bcb5-14e4a7106d0f\") "
Oct 30 03:45:21.948585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.948561    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca\") pod \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\" (UID: \"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb\") "
Oct 30 03:45:21.949325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.949302    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2h9\" (UniqueName: \"kubernetes.io/projected/99050aee-08a4-445e-8f13-1225366e18f0-kube-api-access-sp2h9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.949511 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.949327    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99050aee-08a4-445e-8f13-1225366e18f0-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.949730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.949708    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca" pod "a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" (UID: "a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.950047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.950029    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca" pod "b1178368-e2c2-4e73-bcb5-14e4a7106d0f" (UID: "b1178368-e2c2-4e73-bcb5-14e4a7106d0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:21.963020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.962720    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8" pod "b1178368-e2c2-4e73-bcb5-14e4a7106d0f" (UID: "b1178368-e2c2-4e73-bcb5-14e4a7106d0f"). InnerVolumeSpecName "kube-api-access-t7tp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.963357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:21.963238    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk" pod "a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" (UID: "a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb"). InnerVolumeSpecName "kube-api-access-lmnrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:22.051316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.051277    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.051316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.051315    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmnrk\" (UniqueName: \"kubernetes.io/projected/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-kube-api-access-lmnrk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.051579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.051334    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7tp8\" (UniqueName: \"kubernetes.io/projected/b1178368-e2c2-4e73-bcb5-14e4a7106d0f-kube-api-access-t7tp8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.051579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.051349    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.155048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.155002    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"]
Oct 30 03:45:22.251106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.250781    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:22.308443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.307703    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"
Oct 30 03:45:22.318614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.318582    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf" event={"ID":"99050aee-08a4-445e-8f13-1225366e18f0","Type":"ContainerDied","Data":"3e44a21684347f2b2ea863f83ca683c51bb64989bee91c454c6185d8f6b5b7dd"}
Oct 30 03:45:22.318822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.318811    4877 scope.go:122] "RemoveContainer" containerID="d5e629b05608afca3c98b518261714f9db2e2343b96d26b4586a62f9b6156eb7"
Oct 30 03:45:22.320603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.320516    4877 generic.go:356] "Generic (PLEG): container finished" podID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerID="cbea0c7b21bb8d8d9c2cb5cd7a7ff22bc8d0462459ddf0e270e80233d5e3f1c1" exitCode=60
Oct 30 03:45:22.320709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.320660    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerDied","Data":"cbea0c7b21bb8d8d9c2cb5cd7a7ff22bc8d0462459ddf0e270e80233d5e3f1c1"}
Oct 30 03:45:22.321159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.321137    4877 scope.go:122] "RemoveContainer" containerID="cbea0c7b21bb8d8d9c2cb5cd7a7ff22bc8d0462459ddf0e270e80233d5e3f1c1"
Oct 30 03:45:22.321600 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:22.321572    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc_odh-ai-gateway-infra(09029590-3cdc-4fa7-b239-79c5db99fa0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e"
Oct 30 03:45:22.324245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.323896    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j" event={"ID":"1732eae1-c0ac-4819-804d-b20403ff9d6e","Type":"ContainerDied","Data":"df37cfbc60c22568e3b675dcf2fe2ea19a0d395596441d0c5e793a72fa78264a"}
Oct 30 03:45:22.324245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.324011    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"
Oct 30 03:45:22.330828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.330417    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9" event={"ID":"b1178368-e2c2-4e73-bcb5-14e4a7106d0f","Type":"ContainerDied","Data":"44c85ccd181249e3b842c8456ff5ec323c9b039216fce3426ee7c65df27d7c7e"}
Oct 30 03:45:22.330828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.330563    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"
Oct 30 03:45:22.339270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.337876    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq" event={"ID":"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a","Type":"ContainerDied","Data":"d3cf7539d89fd35ed666b1ec5f4093c0c2516d90fb3824d70dd9ec14f97a5633"}
Oct 30 03:45:22.339270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.337976    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"
Oct 30 03:45:22.339270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.338674    4877 scope.go:122] "RemoveContainer" containerID="beb4cc99dac149a3fe5d64ae1b88e173dff8467141f4c5af0324abd671b505f5"
Oct 30 03:45:22.340646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.340321    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8" event={"ID":"ebe8de4b-ae72-49e7-8cc3-d21faa913e02","Type":"ContainerDied","Data":"5cc71c95dbd24a051cc82f603404ca010c3ef54624916560915d667aee4f584a"}
Oct 30 03:45:22.340646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.340415    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"
Oct 30 03:45:22.355147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.355122    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:22.355717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.355692    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca\") pod \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") "
Oct 30 03:45:22.355805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.355766    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z\" (UniqueName: \"kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z\") pod \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\" (UID: \"f0d2cc44-4d44-4800-af7c-fe1e252b4b3a\") "
Oct 30 03:45:22.357623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.357580    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca" pod "f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" (UID: "f0d2cc44-4d44-4800-af7c-fe1e252b4b3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:22.359209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.359163    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn" event={"ID":"a168bbc2-f646-40d7-897c-0531070a71a5","Type":"ContainerDied","Data":"ce2c33aef87b7716b922fea62a397c08b78ab3889afd5b151068304de1461907"}
Oct 30 03:45:22.359441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.359342    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"
Oct 30 03:45:22.362373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.362351    4877 generic.go:356] "Generic (PLEG): container finished" podID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerID="0e6b35e1a6ac762206350668d348bc4d80e7ed2bd2bd2d4d78c412a312c3b1f6" exitCode=60
Oct 30 03:45:22.362522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.362464    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerDied","Data":"0e6b35e1a6ac762206350668d348bc4d80e7ed2bd2bd2d4d78c412a312c3b1f6"}
Oct 30 03:45:22.363418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.363400    4877 scope.go:122] "RemoveContainer" containerID="0e6b35e1a6ac762206350668d348bc4d80e7ed2bd2bd2d4d78c412a312c3b1f6"
Oct 30 03:45:22.363882 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:22.363857    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g_odh-ai-gateway-infra(97413284-b6ba-44f3-ab43-66f0ff82268b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b"
Oct 30 03:45:22.364783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.364670    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"
Oct 30 03:45:22.364783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.364739    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd" event={"ID":"a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb","Type":"ContainerDied","Data":"9830863a66d9ba0b1fea519a149d5aa1c4a2c67f617a565a28a4a34f500d3584"}
Oct 30 03:45:22.366625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.366417    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7" event={"ID":"6cecb738-c2c3-442e-86c0-a1a6745ca7ff","Type":"ContainerDied","Data":"a2dd8a46c4db47bfe3ff14045b15884fa53ef986f2fcb852b6c17192a9e70ebe"}
Oct 30 03:45:22.366625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.366542    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"
Oct 30 03:45:22.380740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.380719    4877 scope.go:122] "RemoveContainer" containerID="c66104bff627e764a09ffb6bac94c40aa0a1c31da7e3ae9a98effd51c1c85034"
Oct 30 03:45:22.384968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.384938    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z" pod "f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" (UID: "f0d2cc44-4d44-4800-af7c-fe1e252b4b3a"). InnerVolumeSpecName "kube-api-access-hnf7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:22.404186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.403952    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"]
Oct 30 03:45:22.424217 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.424198    4877 scope.go:122] "RemoveContainer" containerID="c253ddcea4b08cfe42d1ec4b07f3dcf096aa600edd7370c5d56f698931d50c85"
Oct 30 03:45:22.440857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.440823    4877 scope.go:122] "RemoveContainer" containerID="08eb07ea515a824786fb84f846fbede9eb0a8b7364b8a4704087003a5c39dbd6"
Oct 30 03:45:22.448828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.448773    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"]
Oct 30 03:45:22.455378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.455325    4877 scope.go:122] "RemoveContainer" containerID="1cec73926b3d58ac11c62462dffdb70900f2fe5d3c78ccb21b84b44bede8577c"
Oct 30 03:45:22.456830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.456771    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca\") pod \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") "
Oct 30 03:45:22.457004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.456919    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2\" (UniqueName: \"kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2\") pod \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\" (UID: \"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8\") "
Oct 30 03:45:22.457231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.457201    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888865-n278j"]
Oct 30 03:45:22.457600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.457579    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca" pod "7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" (UID: "7b0e78d1-c76f-4108-88a3-2fb9158e2bf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:22.458543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.458524    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.458671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.458657    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.458742 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.458673    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnf7z\" (UniqueName: \"kubernetes.io/projected/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a-kube-api-access-hnf7z\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.460010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.459983    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2" pod "7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" (UID: "7b0e78d1-c76f-4108-88a3-2fb9158e2bf8"). InnerVolumeSpecName "kube-api-access-tn6h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:22.587490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.569934    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6h2\" (UniqueName: \"kubernetes.io/projected/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8-kube-api-access-tn6h2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:22.587490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.579157    4877 scope.go:122] "RemoveContainer" containerID="8bbdc039322bb8ff9d748ea98fa58c2a831edcf10d69a1a5f79618f363cc7528"
Oct 30 03:45:22.624219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.624152    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"]
Oct 30 03:45:22.645734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.645713    4877 scope.go:122] "RemoveContainer" containerID="a66110fd5f613a9cdc7c03655e18297a71d9725725bcb7008371499cf23e1cb0"
Oct 30 03:45:22.649572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.649549    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888865-7q6k7"]
Oct 30 03:45:22.668976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.668912    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"]
Oct 30 03:45:22.678806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.678775    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888865-mhdmn"]
Oct 30 03:45:22.699843 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.699816    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"]
Oct 30 03:45:22.701090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.701068    4877 scope.go:122] "RemoveContainer" containerID="043c49a4c0d2fc9eb54851a15f5febd6a4d32fcec2afb1b35a07e73fc64f9efc"
Oct 30 03:45:22.707281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.707250    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888865-wmjmf"]
Oct 30 03:45:22.722079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.722056    4877 scope.go:122] "RemoveContainer" containerID="514016b4e28912addcfa6758f1b052651ccc70d6488f988f2b829fda7aca3df6"
Oct 30 03:45:22.723500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.723480    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:22.739040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.738982    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"]
Oct 30 03:45:22.748111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.748082    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888865-qlqmd"]
Oct 30 03:45:22.910519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.910339    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca\") pod \"38de7398-445d-4e13-9934-6fc61ed7b16b\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") "
Oct 30 03:45:22.910519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.910503    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5\" (UniqueName: \"kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5\") pod \"38de7398-445d-4e13-9934-6fc61ed7b16b\" (UID: \"38de7398-445d-4e13-9934-6fc61ed7b16b\") "
Oct 30 03:45:22.964207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.934760    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca" pod "38de7398-445d-4e13-9934-6fc61ed7b16b" (UID: "38de7398-445d-4e13-9934-6fc61ed7b16b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:22.972542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.972465    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"]
Oct 30 03:45:22.986901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.984537    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888865-qprs9"]
Oct 30 03:45:22.988797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:22.988632    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5" pod "38de7398-445d-4e13-9934-6fc61ed7b16b" (UID: "38de7398-445d-4e13-9934-6fc61ed7b16b"). InnerVolumeSpecName "kube-api-access-87nb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:23.012928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.012894    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38de7398-445d-4e13-9934-6fc61ed7b16b-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.012928 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.012930    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87nb5\" (UniqueName: \"kubernetes.io/projected/38de7398-445d-4e13-9934-6fc61ed7b16b-kube-api-access-87nb5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.046504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.046478    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:23.056271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.056240    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"]
Oct 30 03:45:23.065200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.065170    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888865-qfnd8"]
Oct 30 03:45:23.095674 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.095639    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"]
Oct 30 03:45:23.102644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.102612    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888865-knhhq"]
Oct 30 03:45:23.114138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.114105    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt\" (UniqueName: \"kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt\") pod \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") "
Oct 30 03:45:23.114274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.114150    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca\") pod \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\" (UID: \"56795bbf-5ef2-463b-9bfa-1146e5b91ee0\") "
Oct 30 03:45:23.114570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.114546    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca" pod "56795bbf-5ef2-463b-9bfa-1146e5b91ee0" (UID: "56795bbf-5ef2-463b-9bfa-1146e5b91ee0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:23.114862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.114842    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.116363 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.116338    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt" pod "56795bbf-5ef2-463b-9bfa-1146e5b91ee0" (UID: "56795bbf-5ef2-463b-9bfa-1146e5b91ee0"). InnerVolumeSpecName "kube-api-access-b5hrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:23.216783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.216682    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hrt\" (UniqueName: \"kubernetes.io/projected/56795bbf-5ef2-463b-9bfa-1146e5b91ee0-kube-api-access-b5hrt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.390893 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.390855    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"]
Oct 30 03:45:23.403729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.403645    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h" event={"ID":"7b0e78d1-c76f-4108-88a3-2fb9158e2bf8","Type":"ContainerDied","Data":"347601d3375dacc839f860bd070b50cabfb4efee57c283e01517f84bb9c8b0db"}
Oct 30 03:45:23.403729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.403717    4877 scope.go:122] "RemoveContainer" containerID="21bfb23d9ba2a9c89d1309b3212203bad64d104544aa49af773d5743bde9e73e"
Oct 30 03:45:23.403911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.403663    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"
Oct 30 03:45:23.418758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.418724    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj" event={"ID":"38de7398-445d-4e13-9934-6fc61ed7b16b","Type":"ContainerDied","Data":"32a654497cadf6f69a4003ca76d9c6a0bd5bd77a1c4be63fc722a4ec0e65f2af"}
Oct 30 03:45:23.418969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.418909    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"
Oct 30 03:45:23.430888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.430805    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846" event={"ID":"56795bbf-5ef2-463b-9bfa-1146e5b91ee0","Type":"ContainerDied","Data":"a8f344b05f4835ef4a8b48baac189b5a9eada85a0c3f20e914b1d7600a1d15ce"}
Oct 30 03:45:23.431055 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.431024    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"
Oct 30 03:45:23.441915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.441882    4877 scope.go:122] "RemoveContainer" containerID="0e000a78527766c104b0782f0720d4c51af61e8533b239814e35fe55b7cd10de"
Oct 30 03:45:23.464557 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.464535    4877 scope.go:122] "RemoveContainer" containerID="c5130cbacf5f37dafa6656adf8abaeb266c71bedddb70e0aecf76c8fd5dab153"
Oct 30 03:45:23.529445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.529402    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"]
Oct 30 03:45:23.533383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.533357    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888865-7zg6h"]
Oct 30 03:45:23.555461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.555391    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"]
Oct 30 03:45:23.560336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.560312    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888865-2hnmj"]
Oct 30 03:45:23.603402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.603352    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"]
Oct 30 03:45:23.607102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.607065    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888865-h9846"]
Oct 30 03:45:23.624395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.624363    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:23.685706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.685670    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:23.711169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.711117    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" path="/var/lib/kubelet/pods/1732eae1-c0ac-4819-804d-b20403ff9d6e/volumes"
Oct 30 03:45:23.711878 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.711843    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" path="/var/lib/kubelet/pods/38de7398-445d-4e13-9934-6fc61ed7b16b/volumes"
Oct 30 03:45:23.712462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.712412    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" path="/var/lib/kubelet/pods/56795bbf-5ef2-463b-9bfa-1146e5b91ee0/volumes"
Oct 30 03:45:23.713490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.713460    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" path="/var/lib/kubelet/pods/6cecb738-c2c3-442e-86c0-a1a6745ca7ff/volumes"
Oct 30 03:45:23.714031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.714014    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" path="/var/lib/kubelet/pods/7b0e78d1-c76f-4108-88a3-2fb9158e2bf8/volumes"
Oct 30 03:45:23.714588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.714570    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99050aee-08a4-445e-8f13-1225366e18f0" path="/var/lib/kubelet/pods/99050aee-08a4-445e-8f13-1225366e18f0/volumes"
Oct 30 03:45:23.715501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.715483    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" path="/var/lib/kubelet/pods/a168bbc2-f646-40d7-897c-0531070a71a5/volumes"
Oct 30 03:45:23.716092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.716072    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" path="/var/lib/kubelet/pods/a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb/volumes"
Oct 30 03:45:23.716650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.716632    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" path="/var/lib/kubelet/pods/b1178368-e2c2-4e73-bcb5-14e4a7106d0f/volumes"
Oct 30 03:45:23.717679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.717607    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" path="/var/lib/kubelet/pods/ebe8de4b-ae72-49e7-8cc3-d21faa913e02/volumes"
Oct 30 03:45:23.718182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.718161    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" path="/var/lib/kubelet/pods/f0d2cc44-4d44-4800-af7c-fe1e252b4b3a/volumes"
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.730348    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca\") pod \"97413284-b6ba-44f3-ab43-66f0ff82268b\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") "
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.730400    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca\") pod \"09029590-3cdc-4fa7-b239-79c5db99fa0e\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") "
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.730494    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7\" (UniqueName: \"kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7\") pod \"97413284-b6ba-44f3-ab43-66f0ff82268b\" (UID: \"97413284-b6ba-44f3-ab43-66f0ff82268b\") "
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.730536    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw\" (UniqueName: \"kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw\") pod \"09029590-3cdc-4fa7-b239-79c5db99fa0e\" (UID: \"09029590-3cdc-4fa7-b239-79c5db99fa0e\") "
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.730757    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca" pod "97413284-b6ba-44f3-ab43-66f0ff82268b" (UID: "97413284-b6ba-44f3-ab43-66f0ff82268b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.731079    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca" pod "09029590-3cdc-4fa7-b239-79c5db99fa0e" (UID: "09029590-3cdc-4fa7-b239-79c5db99fa0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:23.732812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.731098    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97413284-b6ba-44f3-ab43-66f0ff82268b-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.734957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.734820    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7" pod "97413284-b6ba-44f3-ab43-66f0ff82268b" (UID: "97413284-b6ba-44f3-ab43-66f0ff82268b"). InnerVolumeSpecName "kube-api-access-fq7v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:23.734957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.734913    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw" pod "09029590-3cdc-4fa7-b239-79c5db99fa0e" (UID: "09029590-3cdc-4fa7-b239-79c5db99fa0e"). InnerVolumeSpecName "kube-api-access-wwxdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:23.832624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.832588    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09029590-3cdc-4fa7-b239-79c5db99fa0e-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.832624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.832621    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7v7\" (UniqueName: \"kubernetes.io/projected/97413284-b6ba-44f3-ab43-66f0ff82268b-kube-api-access-fq7v7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:23.832624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:23.832631    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwxdw\" (UniqueName: \"kubernetes.io/projected/09029590-3cdc-4fa7-b239-79c5db99fa0e-kube-api-access-wwxdw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:24.469739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.469712    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"
Oct 30 03:45:24.470162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.469761    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc" event={"ID":"09029590-3cdc-4fa7-b239-79c5db99fa0e","Type":"ContainerDied","Data":"7ca193c42dd720be445fca595272becd1f5c219d0797799fde0be24bd13dd275"}
Oct 30 03:45:24.470162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.469821    4877 scope.go:122] "RemoveContainer" containerID="cbea0c7b21bb8d8d9c2cb5cd7a7ff22bc8d0462459ddf0e270e80233d5e3f1c1"
Oct 30 03:45:24.474921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.474897    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"
Oct 30 03:45:24.475039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.474904    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g" event={"ID":"97413284-b6ba-44f3-ab43-66f0ff82268b","Type":"ContainerDied","Data":"3a9acc01531c447cb555975b1224710d224ba0957be187712f4866c2346d977c"}
Oct 30 03:45:24.483053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.483033    4877 scope.go:122] "RemoveContainer" containerID="0e6b35e1a6ac762206350668d348bc4d80e7ed2bd2bd2d4d78c412a312c3b1f6"
Oct 30 03:45:24.547623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.547585    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"]
Oct 30 03:45:24.552136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.552103    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888865-lzhqc"]
Oct 30 03:45:24.581299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.581264    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"]
Oct 30 03:45:24.588128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:24.588098    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888865-n7b8g"]
Oct 30 03:45:25.700195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:25.700153    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" path="/var/lib/kubelet/pods/09029590-3cdc-4fa7-b239-79c5db99fa0e/volumes"
Oct 30 03:45:25.700701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:25.700674    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" path="/var/lib/kubelet/pods/97413284-b6ba-44f3-ab43-66f0ff82268b/volumes"
Oct 30 03:45:29.758202 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.758168    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"]
Oct 30 03:45:29.759799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.759768    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.759946 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.759933    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.760029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760019    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:29.760097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760087    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:29.760167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760158    4877 cpu_manager.go:414] "containerMap: removing container" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.760285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760273    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.760367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760357    4877 cpu_manager.go:414] "containerMap: removing container" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:29.760453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760442    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:29.760537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760525    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.760617 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760607    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.760693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760684    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.760758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760749    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.760835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760826    4877 cpu_manager.go:414] "containerMap: removing container" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:29.760900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760890    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:29.760980 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.760972    4877 cpu_manager.go:414] "containerMap: removing container" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.761056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761046    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.761127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761118    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.761191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761182    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.761270 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761261    4877 cpu_manager.go:414] "containerMap: removing container" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.761334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761325    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.761406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761397    4877 cpu_manager.go:414] "containerMap: removing container" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:29.761489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761479    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:29.761562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761553    4877 cpu_manager.go:414] "containerMap: removing container" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.761627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761618    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.761710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761701    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.761774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761766    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.761840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761832    4877 cpu_manager.go:414] "containerMap: removing container" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.761903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761894    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.761976 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.761967    4877 cpu_manager.go:414] "containerMap: removing container" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.762046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762037    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.762116 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762108    4877 cpu_manager.go:414] "containerMap: removing container" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.762188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762178    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.762261 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762252    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:29.762324 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762315    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:29.762394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762386    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:29.762494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762483    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:29.762576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762567    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.762652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762642    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.762720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762711    4877 cpu_manager.go:414] "containerMap: removing container" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.762785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762775    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.762858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762850    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.762924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762915    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.762994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.762985    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.763063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763053    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.763144 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763135    4877 cpu_manager.go:414] "containerMap: removing container" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.763221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763211    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.763298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763290    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:29.763368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763359    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:29.763459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763448    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.763545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763535    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.763621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763612    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:29.763691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763682    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:29.763776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763768    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.763847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763837    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.763932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763922    4877 cpu_manager.go:414] "containerMap: removing container" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.764004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.763995    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.764088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764079    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.764158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764149    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.764244 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764235    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.764315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764304    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.764390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764381    4877 cpu_manager.go:414] "containerMap: removing container" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:29.764486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764473    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:29.764581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764571    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.764652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764643    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.764739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764730    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.764815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764806    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.764888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764879    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.764957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.764948    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.765056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765045    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.765161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765120    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.765161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765141    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:29.765161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765149    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765349    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765363    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765376    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765385    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765397    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765407    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765417    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765443    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765452    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765461    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765472    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765481    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.765490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765492    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765502    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765515    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765524    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765532    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765541    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765547    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765557    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765566    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765577    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765585    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0e78d1-c76f-4108-88a3-2fb9158e2bf8" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765597    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765605    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765614    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765622    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765631    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765642    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765656    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765666    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765675    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765685    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765693    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765704    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:29.766121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.765715    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="09029590-3cdc-4fa7-b239-79c5db99fa0e" containerName="cleanup"
Oct 30 03:45:29.780370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.780341    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.782561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.782531    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"]
Oct 30 03:45:29.794959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.794929    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:45:29.806596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.806575    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:45:29.843154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.843113    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.843290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.843157    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.843336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.843321    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8nv\" (UniqueName: \"kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.944205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.944166    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8nv\" (UniqueName: \"kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.944501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.944369    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.944501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.944417    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.945200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.945175    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.946935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.946916    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:29.963240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:29.963212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8nv\" (UniqueName: \"kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv\") pod \"maas-api-revoke-sub-95911715-hrb44\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:30.100206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:30.100160    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:30.404898 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:45:30.404862    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3a6fc6_2044_48b6_935b_dcb778dd71bb.slice/crio-59b260ff90c856f0dd66e1c079b2939fe17d959b497c4437da4a5bb9d9164216 WatchSource:0}: Error finding container 59b260ff90c856f0dd66e1c079b2939fe17d959b497c4437da4a5bb9d9164216: Status 404 returned error can't find the container with id 59b260ff90c856f0dd66e1c079b2939fe17d959b497c4437da4a5bb9d9164216
Oct 30 03:45:30.407312 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:30.406946    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"]
Oct 30 03:45:30.553343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:30.553296    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44" event={"ID":"2e3a6fc6-2044-48b6-935b-dcb778dd71bb","Type":"ContainerStarted","Data":"59b260ff90c856f0dd66e1c079b2939fe17d959b497c4437da4a5bb9d9164216"}
Oct 30 03:45:31.565691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:31.565649    4877 generic.go:356] "Generic (PLEG): container finished" podID="2e3a6fc6-2044-48b6-935b-dcb778dd71bb" containerID="f362e7bb7c550ddc372872f48739e85eda9552e5d3a718abb99003c59544732e" exitCode=0
Oct 30 03:45:31.566131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:31.565740    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44" event={"ID":"2e3a6fc6-2044-48b6-935b-dcb778dd71bb","Type":"ContainerDied","Data":"f362e7bb7c550ddc372872f48739e85eda9552e5d3a718abb99003c59544732e"}
Oct 30 03:45:32.766722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.766684    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:32.889826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.889736    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca\") pod \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") "
Oct 30 03:45:32.890042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.889848    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca\") pod \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") "
Oct 30 03:45:32.890170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.890144    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv\" (UniqueName: \"kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv\") pod \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\" (UID: \"2e3a6fc6-2044-48b6-935b-dcb778dd71bb\") "
Oct 30 03:45:32.890296 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.890264    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca" pod "2e3a6fc6-2044-48b6-935b-dcb778dd71bb" (UID: "2e3a6fc6-2044-48b6-935b-dcb778dd71bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:32.890763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.890735    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:32.892093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.892061    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca" pod "2e3a6fc6-2044-48b6-935b-dcb778dd71bb" (UID: "2e3a6fc6-2044-48b6-935b-dcb778dd71bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:32.892209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.892187    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv" pod "2e3a6fc6-2044-48b6-935b-dcb778dd71bb" (UID: "2e3a6fc6-2044-48b6-935b-dcb778dd71bb"). InnerVolumeSpecName "kube-api-access-dk8nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:32.992731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.992690    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:32.992731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:32.992726    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8nv\" (UniqueName: \"kubernetes.io/projected/2e3a6fc6-2044-48b6-935b-dcb778dd71bb-kube-api-access-dk8nv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:33.586174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:33.586101    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"
Oct 30 03:45:33.586174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:33.586127    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44" event={"ID":"2e3a6fc6-2044-48b6-935b-dcb778dd71bb","Type":"ContainerDied","Data":"59b260ff90c856f0dd66e1c079b2939fe17d959b497c4437da4a5bb9d9164216"}
Oct 30 03:45:33.586383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:33.586206    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59b260ff90c856f0dd66e1c079b2939fe17d959b497c4437da4a5bb9d9164216"
Oct 30 03:45:41.792016 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.791980    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"]
Oct 30 03:45:41.793601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.793572    4877 cpu_manager.go:414] "containerMap: removing container" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:41.793776 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.793761    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:41.793885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.793874    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:41.793968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.793957    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9eb04b4-4340-4ba9-8c60-af2d4b6bd6bb" containerName="cleanup"
Oct 30 03:45:41.794058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794048    4877 cpu_manager.go:414] "containerMap: removing container" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:41.794142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794132    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:41.794237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794226    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:41.794320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794309    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b186d-1c21-4d11-8ea0-c17cdfd6b739" containerName="cleanup"
Oct 30 03:45:41.794398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794388    4877 cpu_manager.go:414] "containerMap: removing container" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:41.794517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794505    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56795bbf-5ef2-463b-9bfa-1146e5b91ee0" containerName="cleanup"
Oct 30 03:45:41.794633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794621    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:41.794720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794710    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:41.794827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794815    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:41.794914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794902    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:41.795006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.794996    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:41.795092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.795081    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:41.795192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.795181    4877 cpu_manager.go:414] "containerMap: removing container" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:41.795293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.795282    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918ac658-590a-4014-aa4e-0cdb4a173213" containerName="cleanup"
Oct 30 03:45:41.795384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.795374    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2e3a6fc6-2044-48b6-935b-dcb778dd71bb" containerName="revoke-keys"
Oct 30 03:45:41.810596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.810565    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3a6fc6-2044-48b6-935b-dcb778dd71bb" containerName="revoke-keys"
Oct 30 03:45:41.811152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811135    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="97413284-b6ba-44f3-ab43-66f0ff82268b" containerName="cleanup"
Oct 30 03:45:41.811316 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811305    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d2cc44-4d44-4800-af7c-fe1e252b4b3a" containerName="cleanup"
Oct 30 03:45:41.811436 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811413    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a168bbc2-f646-40d7-897c-0531070a71a5" containerName="cleanup"
Oct 30 03:45:41.811548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811538    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1732eae1-c0ac-4819-804d-b20403ff9d6e" containerName="cleanup"
Oct 30 03:45:41.811649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811639    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="38de7398-445d-4e13-9934-6fc61ed7b16b" containerName="cleanup"
Oct 30 03:45:41.811755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811746    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cecb738-c2c3-442e-86c0-a1a6745ca7ff" containerName="cleanup"
Oct 30 03:45:41.811851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811840    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe8de4b-ae72-49e7-8cc3-d21faa913e02" containerName="cleanup"
Oct 30 03:45:41.811967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.811956    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3a6fc6-2044-48b6-935b-dcb778dd71bb" containerName="revoke-keys"
Oct 30 03:45:41.812071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.812061    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="99050aee-08a4-445e-8f13-1225366e18f0" containerName="cleanup"
Oct 30 03:45:41.812179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.812168    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1178368-e2c2-4e73-bcb5-14e4a7106d0f" containerName="cleanup"
Oct 30 03:45:41.828810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.828772    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"]
Oct 30 03:45:41.829226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.829123    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.851107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.851083    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:45:41.855553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.851469    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.856506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.856474    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.856727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.856708    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhbn\" (UniqueName: \"kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.856977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.856903    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:45:41.963331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.963295    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhbn\" (UniqueName: \"kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.963551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.963496    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.963632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.963616    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.964303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.964279    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.965945 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.965925    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:41.982681 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:41.982646    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhbn\" (UniqueName: \"kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn\") pod \"maas-api-revoke-sub-e98fb993-9dbz5\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:42.169732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:42.169650    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:42.473771 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:45:42.473735    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80426dbf_ea7e_436c_8b7d_9c35cb79a5be.slice/crio-a69ba3b0d5da3ec19cad5aacbcf0558c44ecc46708ba397b789bf74557711612 WatchSource:0}: Error finding container a69ba3b0d5da3ec19cad5aacbcf0558c44ecc46708ba397b789bf74557711612: Status 404 returned error can't find the container with id a69ba3b0d5da3ec19cad5aacbcf0558c44ecc46708ba397b789bf74557711612
Oct 30 03:45:42.475957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:42.475935    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"]
Oct 30 03:45:42.668943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:42.668898    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5" event={"ID":"80426dbf-ea7e-436c-8b7d-9c35cb79a5be","Type":"ContainerStarted","Data":"a69ba3b0d5da3ec19cad5aacbcf0558c44ecc46708ba397b789bf74557711612"}
Oct 30 03:45:43.681786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:43.681745    4877 generic.go:356] "Generic (PLEG): container finished" podID="80426dbf-ea7e-436c-8b7d-9c35cb79a5be" containerID="3b19b969fe6c493fd7f696138dfa6925feb0a0450ba3f3f77dd49ec859611a50" exitCode=0
Oct 30 03:45:43.682261 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:43.681794    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5" event={"ID":"80426dbf-ea7e-436c-8b7d-9c35cb79a5be","Type":"ContainerDied","Data":"3b19b969fe6c493fd7f696138dfa6925feb0a0450ba3f3f77dd49ec859611a50"}
Oct 30 03:45:44.886313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:44.886281    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:45.008492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.007877    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn\" (UniqueName: \"kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn\") pod \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") "
Oct 30 03:45:45.008492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.007968    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca\") pod \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") "
Oct 30 03:45:45.008492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.008011    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca\") pod \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\" (UID: \"80426dbf-ea7e-436c-8b7d-9c35cb79a5be\") "
Oct 30 03:45:45.009160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.009118    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca" pod "80426dbf-ea7e-436c-8b7d-9c35cb79a5be" (UID: "80426dbf-ea7e-436c-8b7d-9c35cb79a5be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:45.010981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.010743    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn" pod "80426dbf-ea7e-436c-8b7d-9c35cb79a5be" (UID: "80426dbf-ea7e-436c-8b7d-9c35cb79a5be"). InnerVolumeSpecName "kube-api-access-lrhbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:45.012932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.011356    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca" pod "80426dbf-ea7e-436c-8b7d-9c35cb79a5be" (UID: "80426dbf-ea7e-436c-8b7d-9c35cb79a5be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:45.109993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.109955    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:45.109993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.109988    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrhbn\" (UniqueName: \"kubernetes.io/projected/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-kube-api-access-lrhbn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:45.109993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.109999    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80426dbf-ea7e-436c-8b7d-9c35cb79a5be-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:45:45.706138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.706096    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"
Oct 30 03:45:45.706412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.706154    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5" event={"ID":"80426dbf-ea7e-436c-8b7d-9c35cb79a5be","Type":"ContainerDied","Data":"a69ba3b0d5da3ec19cad5aacbcf0558c44ecc46708ba397b789bf74557711612"}
Oct 30 03:45:45.706412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:45.706218    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a69ba3b0d5da3ec19cad5aacbcf0558c44ecc46708ba397b789bf74557711612"
Oct 30 03:45:46.237392 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:45:46.237355    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:45:55.197356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:55.197315    4877 scope.go:122] "RemoveContainer" containerID="51bce21728bac0d7a8f5b7e2d3fadf71a4cb332015f7d74b500e434c9f2e3767"
Oct 30 03:45:55.209908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:45:55.209888    4877 scope.go:122] "RemoveContainer" containerID="0a3e25735a19e657874efbcb84787ed4c9b5296d71a658ae756b6afe44353119"
Oct 30 03:46:14.697004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.696968    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"]
Oct 30 03:46:14.697953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.697931    4877 cpu_manager.go:414] "containerMap: removing container" podUID="80426dbf-ea7e-436c-8b7d-9c35cb79a5be" containerName="revoke-keys"
Oct 30 03:46:14.697953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.697953    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80426dbf-ea7e-436c-8b7d-9c35cb79a5be" containerName="revoke-keys"
Oct 30 03:46:14.698194 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.698178    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="80426dbf-ea7e-436c-8b7d-9c35cb79a5be" containerName="revoke-keys"
Oct 30 03:46:14.702229 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.702209    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.717726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.717699    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"]
Oct 30 03:46:14.723719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.723694    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:46:14.723869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.723832    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:46:14.803647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.803617    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9wg\" (UniqueName: \"kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.803807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.803776    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.803982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.803956    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.905063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.905030    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9wg\" (UniqueName: \"kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.905230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.905150    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.905275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.905253    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.905983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.905956    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.907631 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.907611    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:14.923892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:14.923868    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9wg\" (UniqueName: \"kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg\") pod \"maas-api-revoke-sub-ec9a34fa-zcpqq\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:15.022619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:15.022595    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:15.329892 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:46:15.329861    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde46bdd6_4804_4996_8929_e8023626beaf.slice/crio-339a688569a466a19f6a1ce75b19daaa940ef6a672af2366435fbbeb944bd217 WatchSource:0}: Error finding container 339a688569a466a19f6a1ce75b19daaa940ef6a672af2366435fbbeb944bd217: Status 404 returned error can't find the container with id 339a688569a466a19f6a1ce75b19daaa940ef6a672af2366435fbbeb944bd217
Oct 30 03:46:15.332856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:15.332826    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"]
Oct 30 03:46:16.011797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.011701    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq" event={"ID":"de46bdd6-4804-4996-8929-e8023626beaf","Type":"ContainerStarted","Data":"be2d952a303709cb44422ebc6fa05edc9299cda1dbc484e942565f987c7cd976"}
Oct 30 03:46:16.011797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.011763    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq" event={"ID":"de46bdd6-4804-4996-8929-e8023626beaf","Type":"ContainerStarted","Data":"339a688569a466a19f6a1ce75b19daaa940ef6a672af2366435fbbeb944bd217"}
Oct 30 03:46:16.236815 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:46:16.236769    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:46:16.651025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.650894    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq" podStartSLOduration=2.650877008 podStartE2EDuration="2.650877008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:16.073021239 +0000 UTC m=+2604.501910658" watchObservedRunningTime="2026-10-30 03:46:16.650877008 +0000 UTC m=+2605.079766429"
Oct 30 03:46:16.655977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.655953    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"]
Oct 30 03:46:16.660653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.660635    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.679171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.679145    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"]
Oct 30 03:46:16.735293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.735260    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.735514 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.735320    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.735586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.735511    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcgn\" (UniqueName: \"kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.837045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.837009    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcgn\" (UniqueName: \"kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.837242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.837169    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.837242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.837223    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.837963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.837931    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.839613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.839582    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.855469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.855419    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcgn\" (UniqueName: \"kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn\") pod \"maas-api-revoke-sub-5204ed74-smlkt\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:16.979886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:16.979807    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:17.023654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:17.023619    4877 generic.go:356] "Generic (PLEG): container finished" podID="de46bdd6-4804-4996-8929-e8023626beaf" containerID="be2d952a303709cb44422ebc6fa05edc9299cda1dbc484e942565f987c7cd976" exitCode=0
Oct 30 03:46:17.024139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:17.023701    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq" event={"ID":"de46bdd6-4804-4996-8929-e8023626beaf","Type":"ContainerDied","Data":"be2d952a303709cb44422ebc6fa05edc9299cda1dbc484e942565f987c7cd976"}
Oct 30 03:46:17.299560 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:46:17.299521    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c32834_13e1_46ea_bb9f_ff6bbd3a5bb0.slice/crio-f53d4a195db95a1cab1fe9a82168179aabf5a6e367033cfc48da76d9d572c7bc WatchSource:0}: Error finding container f53d4a195db95a1cab1fe9a82168179aabf5a6e367033cfc48da76d9d572c7bc: Status 404 returned error can't find the container with id f53d4a195db95a1cab1fe9a82168179aabf5a6e367033cfc48da76d9d572c7bc
Oct 30 03:46:17.301755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:17.301736    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"]
Oct 30 03:46:18.034601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.034555    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt" event={"ID":"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0","Type":"ContainerStarted","Data":"7f9e111aa21dda08c6e30e703cd1692cb0de77ac8e00989d663d5ecf5b52b137"}
Oct 30 03:46:18.035065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.034622    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt" event={"ID":"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0","Type":"ContainerStarted","Data":"f53d4a195db95a1cab1fe9a82168179aabf5a6e367033cfc48da76d9d572c7bc"}
Oct 30 03:46:18.076011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.075953    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt" podStartSLOduration=2.075938001 podStartE2EDuration="2.075938001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:18.070567481 +0000 UTC m=+2606.499456899" watchObservedRunningTime="2026-10-30 03:46:18.075938001 +0000 UTC m=+2606.504827453"
Oct 30 03:46:18.215680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.215652    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:18.260479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.254220    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca\") pod \"de46bdd6-4804-4996-8929-e8023626beaf\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") "
Oct 30 03:46:18.260479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.256127    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca\") pod \"de46bdd6-4804-4996-8929-e8023626beaf\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") "
Oct 30 03:46:18.260479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.256484    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca" pod "de46bdd6-4804-4996-8929-e8023626beaf" (UID: "de46bdd6-4804-4996-8929-e8023626beaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:18.260479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.256678    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg\" (UniqueName: \"kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg\") pod \"de46bdd6-4804-4996-8929-e8023626beaf\" (UID: \"de46bdd6-4804-4996-8929-e8023626beaf\") "
Oct 30 03:46:18.260479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.257350    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:18.263213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.263183    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg" pod "de46bdd6-4804-4996-8929-e8023626beaf" (UID: "de46bdd6-4804-4996-8929-e8023626beaf"). InnerVolumeSpecName "kube-api-access-fv9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:18.267610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.267581    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca" pod "de46bdd6-4804-4996-8929-e8023626beaf" (UID: "de46bdd6-4804-4996-8929-e8023626beaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:18.358406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.358377    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de46bdd6-4804-4996-8929-e8023626beaf-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:18.358406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:18.358406    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9wg\" (UniqueName: \"kubernetes.io/projected/de46bdd6-4804-4996-8929-e8023626beaf-kube-api-access-fv9wg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:19.043921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:19.043882    4877 generic.go:356] "Generic (PLEG): container finished" podID="07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" containerID="7f9e111aa21dda08c6e30e703cd1692cb0de77ac8e00989d663d5ecf5b52b137" exitCode=0
Oct 30 03:46:19.044401 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:19.043968    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt" event={"ID":"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0","Type":"ContainerDied","Data":"7f9e111aa21dda08c6e30e703cd1692cb0de77ac8e00989d663d5ecf5b52b137"}
Oct 30 03:46:19.045474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:19.045438    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"
Oct 30 03:46:19.045474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:19.045459    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq" event={"ID":"de46bdd6-4804-4996-8929-e8023626beaf","Type":"ContainerDied","Data":"339a688569a466a19f6a1ce75b19daaa940ef6a672af2366435fbbeb944bd217"}
Oct 30 03:46:19.045638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:19.045493    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339a688569a466a19f6a1ce75b19daaa940ef6a672af2366435fbbeb944bd217"
Oct 30 03:46:20.235551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.235525    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:20.286283 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.286247    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca\") pod \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") "
Oct 30 03:46:20.286521 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.286452    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn\" (UniqueName: \"kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn\") pod \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") "
Oct 30 03:46:20.287336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.286642    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca\") pod \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\" (UID: \"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0\") "
Oct 30 03:46:20.287336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.286980    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca" pod "07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" (UID: "07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:20.287336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.287235    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:20.290173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.290133    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca" pod "07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" (UID: "07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:20.290512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.290474    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn" pod "07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" (UID: "07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0"). InnerVolumeSpecName "kube-api-access-vqcgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:20.388737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.388702    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:20.388737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:20.388732    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqcgn\" (UniqueName: \"kubernetes.io/projected/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0-kube-api-access-vqcgn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:21.074287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:21.074249    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt" event={"ID":"07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0","Type":"ContainerDied","Data":"f53d4a195db95a1cab1fe9a82168179aabf5a6e367033cfc48da76d9d572c7bc"}
Oct 30 03:46:21.074470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:21.074328    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f53d4a195db95a1cab1fe9a82168179aabf5a6e367033cfc48da76d9d572c7bc"
Oct 30 03:46:21.074470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:21.074257    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"
Oct 30 03:46:46.243179 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:46:46.243132    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:46:47.746241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.746205    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"]
Oct 30 03:46:47.746901 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.746885    4877 cpu_manager.go:414] "containerMap: removing container" podUID="de46bdd6-4804-4996-8929-e8023626beaf" containerName="revoke-keys"
Oct 30 03:46:47.746947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.746902    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de46bdd6-4804-4996-8929-e8023626beaf" containerName="revoke-keys"
Oct 30 03:46:47.746947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.746931    4877 cpu_manager.go:414] "containerMap: removing container" podUID="07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" containerName="revoke-keys"
Oct 30 03:46:47.746947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.746937    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" containerName="revoke-keys"
Oct 30 03:46:47.747092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.747082    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" containerName="revoke-keys"
Oct 30 03:46:47.747128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.747102    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="de46bdd6-4804-4996-8929-e8023626beaf" containerName="revoke-keys"
Oct 30 03:46:47.751149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.751131    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.767462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.767411    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"]
Oct 30 03:46:47.773467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.773411    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:46:47.773657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.773628    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:46:47.863771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.863742    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.863963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.863937    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhrk\" (UniqueName: \"kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.864080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.864060    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.965829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.965795    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.965993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.965852    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.966038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.966004    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhrk\" (UniqueName: \"kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.966594 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.966572    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.968227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.968206    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:47.984378 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:47.984356    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhrk\" (UniqueName: \"kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk\") pod \"maas-api-revoke-sub-39da9501-7dm25\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:48.068970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:48.068929    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:48.393477 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:46:48.393409    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c255e4_a5f3_40d6_9742_13ae33d599ee.slice/crio-fc6d90d97b95c3a631cd5cb5855208d613345a067da1e3039f4ea3692c55bfa2 WatchSource:0}: Error finding container fc6d90d97b95c3a631cd5cb5855208d613345a067da1e3039f4ea3692c55bfa2: Status 404 returned error can't find the container with id fc6d90d97b95c3a631cd5cb5855208d613345a067da1e3039f4ea3692c55bfa2
Oct 30 03:46:48.397135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:48.397104    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"]
Oct 30 03:46:48.491824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:48.491791    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25" event={"ID":"08c255e4-a5f3-40d6-9742-13ae33d599ee","Type":"ContainerStarted","Data":"fc6d90d97b95c3a631cd5cb5855208d613345a067da1e3039f4ea3692c55bfa2"}
Oct 30 03:46:49.501532 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:49.501487    4877 generic.go:356] "Generic (PLEG): container finished" podID="08c255e4-a5f3-40d6-9742-13ae33d599ee" containerID="ed972cbd6621adc99efc4c8f44c8c9203650dbf633e717ec383f4d983a0471b9" exitCode=0
Oct 30 03:46:49.501951 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:49.501570    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25" event={"ID":"08c255e4-a5f3-40d6-9742-13ae33d599ee","Type":"ContainerDied","Data":"ed972cbd6621adc99efc4c8f44c8c9203650dbf633e717ec383f4d983a0471b9"}
Oct 30 03:46:50.688221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.688196    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:50.840613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.840581    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca\") pod \"08c255e4-a5f3-40d6-9742-13ae33d599ee\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") "
Oct 30 03:46:50.840794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.840731    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca\") pod \"08c255e4-a5f3-40d6-9742-13ae33d599ee\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") "
Oct 30 03:46:50.840794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.840779    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk\" (UniqueName: \"kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk\") pod \"08c255e4-a5f3-40d6-9742-13ae33d599ee\" (UID: \"08c255e4-a5f3-40d6-9742-13ae33d599ee\") "
Oct 30 03:46:50.841130 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.841095    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca" pod "08c255e4-a5f3-40d6-9742-13ae33d599ee" (UID: "08c255e4-a5f3-40d6-9742-13ae33d599ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:50.842047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.842024    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.843125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.843100    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca" pod "08c255e4-a5f3-40d6-9742-13ae33d599ee" (UID: "08c255e4-a5f3-40d6-9742-13ae33d599ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:50.843213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.843176    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk" pod "08c255e4-a5f3-40d6-9742-13ae33d599ee" (UID: "08c255e4-a5f3-40d6-9742-13ae33d599ee"). InnerVolumeSpecName "kube-api-access-pfhrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:50.947449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.945852    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhrk\" (UniqueName: \"kubernetes.io/projected/08c255e4-a5f3-40d6-9742-13ae33d599ee-kube-api-access-pfhrk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:50.947449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:50.945893    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c255e4-a5f3-40d6-9742-13ae33d599ee-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:46:51.527189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:51.527146    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25" event={"ID":"08c255e4-a5f3-40d6-9742-13ae33d599ee","Type":"ContainerDied","Data":"fc6d90d97b95c3a631cd5cb5855208d613345a067da1e3039f4ea3692c55bfa2"}
Oct 30 03:46:51.527189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:51.527190    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"
Oct 30 03:46:51.527463 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:51.527230    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6d90d97b95c3a631cd5cb5855208d613345a067da1e3039f4ea3692c55bfa2"
Oct 30 03:46:55.338037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:55.338001    4877 scope.go:122] "RemoveContainer" containerID="a84435a59716e8a0038389f64a09d2cb85c62ce41b298f306d06ea5bd683c858"
Oct 30 03:46:55.353826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:55.353802    4877 scope.go:122] "RemoveContainer" containerID="1c87c31920a3975078e5d9864a43f73f98644dd13370b2096b5a8bd417356d40"
Oct 30 03:46:55.363848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:55.363829    4877 scope.go:122] "RemoveContainer" containerID="4f18360192367a251c448a262eb8466ee6c4575a9273bc1cc8ddea6b19d06f90"
Oct 30 03:46:55.373939 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:46:55.373918    4877 scope.go:122] "RemoveContainer" containerID="bafee640f55e21707cb0a8abee29b8afc6c13117243d831e6159b870209248a0"
Oct 30 03:47:06.047558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.047479    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"]
Oct 30 03:47:06.048264 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.048246    4877 cpu_manager.go:414] "containerMap: removing container" podUID="08c255e4-a5f3-40d6-9742-13ae33d599ee" containerName="revoke-keys"
Oct 30 03:47:06.048310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.048265    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c255e4-a5f3-40d6-9742-13ae33d599ee" containerName="revoke-keys"
Oct 30 03:47:06.048459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.048419    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c255e4-a5f3-40d6-9742-13ae33d599ee" containerName="revoke-keys"
Oct 30 03:47:06.053982 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.053960    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.070572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.070546    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"]
Oct 30 03:47:06.076969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.076947    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\""
Oct 30 03:47:06.077290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.077272    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\""
Oct 30 03:47:06.077457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.077368    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-df757712a33a\""
Oct 30 03:47:06.077640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.077622    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-mpmb2\""
Oct 30 03:47:06.079968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.079946    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\""
Oct 30 03:47:06.080092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.080009    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\""
Oct 30 03:47:06.178818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.178783    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.179008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.178918    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dgz\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.179008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.178970    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.179162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.179144    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.284380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.284329    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.285693 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.285664    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.285932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.285914    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dgz\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.286053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.286036    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.295481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.285442    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.302501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.301139    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.310667 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.310636    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.342443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.341574    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dgz\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.375448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.374556    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.768105 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:47:06.768076    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e83bd0_4820_4f38_8f86_d1ff3e88262c.slice/crio-19a2d9e848412bc18aa1c32133469e9a4adcc5f18b40b6d7bbfdc768020c5edb WatchSource:0}: Error finding container 19a2d9e848412bc18aa1c32133469e9a4adcc5f18b40b6d7bbfdc768020c5edb: Status 404 returned error can't find the container with id 19a2d9e848412bc18aa1c32133469e9a4adcc5f18b40b6d7bbfdc768020c5edb
Oct 30 03:47:06.771633 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.771605    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"]
Oct 30 03:47:06.940841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.940801    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz" event={"ID":"b2e83bd0-4820-4f38-8f86-d1ff3e88262c","Type":"ContainerStarted","Data":"1d02c1b2d23e09632864c1caccce1caedcc8a40a0249ced7ec71b2da2ab714bb"}
Oct 30 03:47:06.943009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.942985    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:06.943191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.943175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz" event={"ID":"b2e83bd0-4820-4f38-8f86-d1ff3e88262c","Type":"ContainerStarted","Data":"19a2d9e848412bc18aa1c32133469e9a4adcc5f18b40b6d7bbfdc768020c5edb"}
Oct 30 03:47:06.981132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:06.981077    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz" podStartSLOduration=0.981062802 podStartE2EDuration="981.062802ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:06.979492242 +0000 UTC m=+2655.408381671" watchObservedRunningTime="2026-10-30 03:47:06.981062802 +0000 UTC m=+2655.409952220"
Oct 30 03:47:08.267564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.267531    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"]
Oct 30 03:47:08.273576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.273548    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.293248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.293213    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"]
Oct 30 03:47:08.299436 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.299391    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"]
Oct 30 03:47:08.305555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.305532    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.333469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.333443    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"]
Oct 30 03:47:08.441546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441508    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.441724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441581    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.441724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441612    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dll\" (UniqueName: \"kubernetes.io/projected/11ff9767-6324-495f-be0e-6bca50031925-kube-api-access-s7dll\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.441724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441676    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7p7\" (UniqueName: \"kubernetes.io/projected/1057196a-2b55-42a3-a8d4-4918a927d4b7-kube-api-access-rl7p7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.441724 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441703    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ff9767-6324-495f-be0e-6bca50031925-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.441962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441728    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.441962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441778    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.441962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441842    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.441962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441895    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.441962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441930    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9767-6324-495f-be0e-6bca50031925-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.442205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.441998    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.442205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442070    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.442205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442160    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ff9767-6324-495f-be0e-6bca50031925-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.442328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442210    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1057196a-2b55-42a3-a8d4-4918a927d4b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.442328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442252    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.442328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442271    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.442328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442314    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.442574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.442364    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.543747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.543709    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.543938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.543754    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.543938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.543781    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.543938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.543819    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.543938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.543923    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.543966    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544001    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dll\" (UniqueName: \"kubernetes.io/projected/11ff9767-6324-495f-be0e-6bca50031925-kube-api-access-s7dll\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544047    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7p7\" (UniqueName: \"kubernetes.io/projected/1057196a-2b55-42a3-a8d4-4918a927d4b7-kube-api-access-rl7p7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544084    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ff9767-6324-495f-be0e-6bca50031925-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544122    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544155    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544179    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544202    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544234    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544268    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9767-6324-495f-be0e-6bca50031925-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544300    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544350    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544383    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544441    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544470    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ff9767-6324-495f-be0e-6bca50031925-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.544601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544550    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1057196a-2b55-42a3-a8d4-4918a927d4b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.545191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544718    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.545191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544301    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.545191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544719    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.545191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.544832    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.545191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.545019    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.545503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.545324    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1057196a-2b55-42a3-a8d4-4918a927d4b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.545503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.545369    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ff9767-6324-495f-be0e-6bca50031925-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.546513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.546486    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ff9767-6324-495f-be0e-6bca50031925-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.547015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.546992    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.547085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.547064    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.547413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.547391    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ff9767-6324-495f-be0e-6bca50031925-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.563114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.563091    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ff9767-6324-495f-be0e-6bca50031925-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.563462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.563405    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1057196a-2b55-42a3-a8d4-4918a927d4b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.563580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.563546    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dll\" (UniqueName: \"kubernetes.io/projected/11ff9767-6324-495f-be0e-6bca50031925-kube-api-access-s7dll\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fs5rj\" (UID: \"11ff9767-6324-495f-be0e-6bca50031925\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.563875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.563854    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7p7\" (UniqueName: \"kubernetes.io/projected/1057196a-2b55-42a3-a8d4-4918a927d4b7-kube-api-access-rl7p7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qzl6k\" (UID: \"1057196a-2b55-42a3-a8d4-4918a927d4b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.595089 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.595064    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:08.638098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.638070    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:08.920861 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:47:08.920821    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ff9767_6324_495f_be0e_6bca50031925.slice/crio-e98c43c6c339bbd0bfe099aef9b0194f35985b1d0cc0b9e5ab7eb548a829653e WatchSource:0}: Error finding container e98c43c6c339bbd0bfe099aef9b0194f35985b1d0cc0b9e5ab7eb548a829653e: Status 404 returned error can't find the container with id e98c43c6c339bbd0bfe099aef9b0194f35985b1d0cc0b9e5ab7eb548a829653e
Oct 30 03:47:08.924048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.923999    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"]
Oct 30 03:47:08.927076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.927041    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:08.927189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.927111    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:08.927189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.927141    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:08.963153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.963022    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" event={"ID":"11ff9767-6324-495f-be0e-6bca50031925","Type":"ContainerStarted","Data":"e98c43c6c339bbd0bfe099aef9b0194f35985b1d0cc0b9e5ab7eb548a829653e"}
Oct 30 03:47:08.966865 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:47:08.966840    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1057196a_2b55_42a3_a8d4_4918a927d4b7.slice/crio-a3350a5dde2efab910f1fa3e132a49270024beac7149916e32f9cf90166a9135 WatchSource:0}: Error finding container a3350a5dde2efab910f1fa3e132a49270024beac7149916e32f9cf90166a9135: Status 404 returned error can't find the container with id a3350a5dde2efab910f1fa3e132a49270024beac7149916e32f9cf90166a9135
Oct 30 03:47:08.969526 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.969492    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:08.969623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.969552    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"]
Oct 30 03:47:08.969623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.969561    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:08.969702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:08.969645    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:09.972638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:09.972596    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" event={"ID":"11ff9767-6324-495f-be0e-6bca50031925","Type":"ContainerStarted","Data":"e17b8691e84d85ad1be714f627742ec7d5fdf53dadd22f9fe7389e61905356df"}
Oct 30 03:47:09.974825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:09.974794    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" event={"ID":"1057196a-2b55-42a3-a8d4-4918a927d4b7","Type":"ContainerStarted","Data":"ef059ba4f301bad16b3b378357f409e602992b1f1a106422957c311ceb997389"}
Oct 30 03:47:09.974947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:09.974849    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" event={"ID":"1057196a-2b55-42a3-a8d4-4918a927d4b7","Type":"ContainerStarted","Data":"a3350a5dde2efab910f1fa3e132a49270024beac7149916e32f9cf90166a9135"}
Oct 30 03:47:10.048319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.048240    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podStartSLOduration=2.048222701 podStartE2EDuration="2.048222701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:10.036778067 +0000 UTC m=+2658.465667486" watchObservedRunningTime="2026-10-30 03:47:10.048222701 +0000 UTC m=+2658.477112110"
Oct 30 03:47:10.103447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.102524    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podStartSLOduration=2.102501889 podStartE2EDuration="2.102501889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:10.075409587 +0000 UTC m=+2658.504299005" watchObservedRunningTime="2026-10-30 03:47:10.102501889 +0000 UTC m=+2658.531391310"
Oct 30 03:47:10.602888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.602853    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:10.603265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.603243    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:10.603461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.603413    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:10.641446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.640977    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:10.642788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.642764    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:10.642968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:10.642943    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:11.607900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:11.607862    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:11.620558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:11.620512    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:11.648503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:11.648464    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:11.648787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:11.648752    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:12.607447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:12.605413    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:12.607447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:12.605498    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:12.640905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:12.640843    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:12.641460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:12.640913    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:13.602448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:13.599528    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:13.602448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:13.599601    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:13.640450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:13.640360    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:13.640632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:13.640451    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:14.608448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:14.605862    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:14.608448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:14.605925    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:14.639921 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:14.639887    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:14.640233 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:14.640189    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:15.605448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:15.598616    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:15.605448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:15.598705    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:15.640931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:15.640310    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:15.640931 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:15.640371    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:16.249064 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:47:16.249021    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:47:16.596830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:16.596794    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:16.597167 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:16.597135    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:16.640251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:16.639817    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:16.640251 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:16.639877    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:17.608272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:17.597822    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:17.608272 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:17.597884    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:17.640929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:17.640894    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 03:47:17.641195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:17.641170    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 03:47:17.986049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:17.985969    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:18.603446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:18.597139    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 03:47:18.603446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:18.597198    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 03:47:19.681457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:19.680539    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:47:19.681457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:19.680630    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:47:19.828480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:19.828019    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:19.832875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:19.832851    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:19.834806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:19.834784    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k"
Oct 30 03:47:19.972200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:19.972121    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"]
Oct 30 03:47:20.000590 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.000562    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.021078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.018767    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:47:20.074908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.074874    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:47:20.081919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.081885    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"]
Oct 30 03:47:20.119762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.117703    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.119762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.117829    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7fd\" (UniqueName: \"kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.119762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.117922    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.221446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.221330    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7fd\" (UniqueName: \"kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.221636 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.221461    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.227170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.221713    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.227170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.223156    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.240451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.232282    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.247451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.246063    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7fd\" (UniqueName: \"kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd\") pod \"maas-api-revoke-sub-47f2a4f0-sb5j6\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.271191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.270090    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:20.271191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.270625    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:20.271545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.271523    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj"
Oct 30 03:47:20.389446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.388676    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:20.786558 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:47:20.782795    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f6d363_edf0_477b_b114_6d80b7d192b5.slice/crio-12d5166cc1945c1a712ed87edce4f9cb6df880100f2919e12645cd55d2f019a8 WatchSource:0}: Error finding container 12d5166cc1945c1a712ed87edce4f9cb6df880100f2919e12645cd55d2f019a8: Status 404 returned error can't find the container with id 12d5166cc1945c1a712ed87edce4f9cb6df880100f2919e12645cd55d2f019a8
Oct 30 03:47:20.786558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.786184    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"]
Oct 30 03:47:20.923157 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.923117    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"]
Oct 30 03:47:20.930573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.930548    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:20.961892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:20.961839    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"]
Oct 30 03:47:21.033027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.032988    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wx7f\" (UniqueName: \"kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.033205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.033159    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.033205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.033194    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.135185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.135073    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.135185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.135162    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.135488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.135464    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wx7f\" (UniqueName: \"kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.135948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.135918    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.137999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.137973    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.154478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.154411    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wx7f\" (UniqueName: \"kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f\") pod \"maas-api-revoke-sub-97066d73-vbwp8\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.250586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.250553    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:21.554189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.550085    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6" event={"ID":"46f6d363-edf0-477b-b114-6d80b7d192b5","Type":"ContainerStarted","Data":"b126087b05a153736f1dbf19c4f4bc59dfd8c0ef91795c65e88f658fcaeb82a1"}
Oct 30 03:47:21.554189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.550148    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6" event={"ID":"46f6d363-edf0-477b-b114-6d80b7d192b5","Type":"ContainerStarted","Data":"12d5166cc1945c1a712ed87edce4f9cb6df880100f2919e12645cd55d2f019a8"}
Oct 30 03:47:21.649556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.648917    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6" podStartSLOduration=2.648896953 podStartE2EDuration="2.648896953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:21.62450503 +0000 UTC m=+2670.053394449" watchObservedRunningTime="2026-10-30 03:47:21.648896953 +0000 UTC m=+2670.077786374"
Oct 30 03:47:21.659534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:21.659510    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"]
Oct 30 03:47:22.561452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:22.561390    4877 generic.go:356] "Generic (PLEG): container finished" podID="46f6d363-edf0-477b-b114-6d80b7d192b5" containerID="b126087b05a153736f1dbf19c4f4bc59dfd8c0ef91795c65e88f658fcaeb82a1" exitCode=0
Oct 30 03:47:22.561833 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:22.561452    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6" event={"ID":"46f6d363-edf0-477b-b114-6d80b7d192b5","Type":"ContainerDied","Data":"b126087b05a153736f1dbf19c4f4bc59dfd8c0ef91795c65e88f658fcaeb82a1"}
Oct 30 03:47:22.563235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:22.563208    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8" event={"ID":"5f315c41-cf78-483e-9d57-2237b2d9ee12","Type":"ContainerStarted","Data":"9fa77fcc265ed130e058497f025dcc5b21fc98a774158b315ba21a184da0d735"}
Oct 30 03:47:22.563402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:22.563390    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8" event={"ID":"5f315c41-cf78-483e-9d57-2237b2d9ee12","Type":"ContainerStarted","Data":"0f801388d8b4e5561a231116b9461c6d74474da61b7d8ecaf97c7911533b0d31"}
Oct 30 03:47:22.643446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:22.642465    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8" podStartSLOduration=2.642445828 podStartE2EDuration="2.642445828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:22.640066891 +0000 UTC m=+2671.068956331" watchObservedRunningTime="2026-10-30 03:47:22.642445828 +0000 UTC m=+2671.071335248"
Oct 30 03:47:23.573888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.573850    4877 generic.go:356] "Generic (PLEG): container finished" podID="5f315c41-cf78-483e-9d57-2237b2d9ee12" containerID="9fa77fcc265ed130e058497f025dcc5b21fc98a774158b315ba21a184da0d735" exitCode=0
Oct 30 03:47:23.574327 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.573932    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8" event={"ID":"5f315c41-cf78-483e-9d57-2237b2d9ee12","Type":"ContainerDied","Data":"9fa77fcc265ed130e058497f025dcc5b21fc98a774158b315ba21a184da0d735"}
Oct 30 03:47:23.757453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.757408    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:23.811528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.811490    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca\") pod \"46f6d363-edf0-477b-b114-6d80b7d192b5\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") "
Oct 30 03:47:23.811717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.811585    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca\") pod \"46f6d363-edf0-477b-b114-6d80b7d192b5\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") "
Oct 30 03:47:23.811717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.811620    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd\" (UniqueName: \"kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd\") pod \"46f6d363-edf0-477b-b114-6d80b7d192b5\" (UID: \"46f6d363-edf0-477b-b114-6d80b7d192b5\") "
Oct 30 03:47:23.813673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.813637    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca" pod "46f6d363-edf0-477b-b114-6d80b7d192b5" (UID: "46f6d363-edf0-477b-b114-6d80b7d192b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:23.815955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.815927    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca" pod "46f6d363-edf0-477b-b114-6d80b7d192b5" (UID: "46f6d363-edf0-477b-b114-6d80b7d192b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:23.816071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.816053    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd" pod "46f6d363-edf0-477b-b114-6d80b7d192b5" (UID: "46f6d363-edf0-477b-b114-6d80b7d192b5"). InnerVolumeSpecName "kube-api-access-jw7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:23.913580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.913544    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:23.913580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.913575    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46f6d363-edf0-477b-b114-6d80b7d192b5-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:23.913580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:23.913584    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7fd\" (UniqueName: \"kubernetes.io/projected/46f6d363-edf0-477b-b114-6d80b7d192b5-kube-api-access-jw7fd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:24.585543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.585513    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"
Oct 30 03:47:24.586004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.585548    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6" event={"ID":"46f6d363-edf0-477b-b114-6d80b7d192b5","Type":"ContainerDied","Data":"12d5166cc1945c1a712ed87edce4f9cb6df880100f2919e12645cd55d2f019a8"}
Oct 30 03:47:24.586004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.585615    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d5166cc1945c1a712ed87edce4f9cb6df880100f2919e12645cd55d2f019a8"
Oct 30 03:47:24.779294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.779262    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:24.929126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.929031    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca\") pod \"5f315c41-cf78-483e-9d57-2237b2d9ee12\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") "
Oct 30 03:47:24.929126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.929106    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca\") pod \"5f315c41-cf78-483e-9d57-2237b2d9ee12\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") "
Oct 30 03:47:24.929382 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.929226    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f\" (UniqueName: \"kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f\") pod \"5f315c41-cf78-483e-9d57-2237b2d9ee12\" (UID: \"5f315c41-cf78-483e-9d57-2237b2d9ee12\") "
Oct 30 03:47:24.929560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.929535    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca" pod "5f315c41-cf78-483e-9d57-2237b2d9ee12" (UID: "5f315c41-cf78-483e-9d57-2237b2d9ee12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:24.929949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.929929    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:24.931460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.931413    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca" pod "5f315c41-cf78-483e-9d57-2237b2d9ee12" (UID: "5f315c41-cf78-483e-9d57-2237b2d9ee12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:24.931587 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:24.931565    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f" pod "5f315c41-cf78-483e-9d57-2237b2d9ee12" (UID: "5f315c41-cf78-483e-9d57-2237b2d9ee12"). InnerVolumeSpecName "kube-api-access-9wx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:25.031107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:25.031073    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wx7f\" (UniqueName: \"kubernetes.io/projected/5f315c41-cf78-483e-9d57-2237b2d9ee12-kube-api-access-9wx7f\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:25.031107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:25.031101    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f315c41-cf78-483e-9d57-2237b2d9ee12-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:25.505875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:25.505834    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"]
Oct 30 03:47:25.598646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:25.598614    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"
Oct 30 03:47:25.598646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:25.598631    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8" event={"ID":"5f315c41-cf78-483e-9d57-2237b2d9ee12","Type":"ContainerDied","Data":"0f801388d8b4e5561a231116b9461c6d74474da61b7d8ecaf97c7911533b0d31"}
Oct 30 03:47:25.599100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:25.598697    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f801388d8b4e5561a231116b9461c6d74474da61b7d8ecaf97c7911533b0d31"
Oct 30 03:47:30.506510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.506465    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz" podUID="b2e83bd0-4820-4f38-8f86-d1ff3e88262c" containerName="payload-processing" containerID="cri-o://1d02c1b2d23e09632864c1caccce1caedcc8a40a0249ced7ec71b2da2ab714bb" gracePeriod=25
Oct 30 03:47:30.657396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.657354    4877 generic.go:356] "Generic (PLEG): container finished" podID="b2e83bd0-4820-4f38-8f86-d1ff3e88262c" containerID="1d02c1b2d23e09632864c1caccce1caedcc8a40a0249ced7ec71b2da2ab714bb" exitCode=0
Oct 30 03:47:30.657574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.657437    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz" event={"ID":"b2e83bd0-4820-4f38-8f86-d1ff3e88262c","Type":"ContainerDied","Data":"1d02c1b2d23e09632864c1caccce1caedcc8a40a0249ced7ec71b2da2ab714bb"}
Oct 30 03:47:30.813770 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.813749    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:30.919591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.919557    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz\") pod \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") "
Oct 30 03:47:30.919793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.919663    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume\") pod \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") "
Oct 30 03:47:30.919793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.919694    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume\") pod \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") "
Oct 30 03:47:30.919793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.919766    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials\") pod \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\" (UID: \"b2e83bd0-4820-4f38-8f86-d1ff3e88262c\") "
Oct 30 03:47:30.920062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.920032    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume" pod "b2e83bd0-4820-4f38-8f86-d1ff3e88262c" (UID: "b2e83bd0-4820-4f38-8f86-d1ff3e88262c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:30.920200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.920173    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume" pod "b2e83bd0-4820-4f38-8f86-d1ff3e88262c" (UID: "b2e83bd0-4820-4f38-8f86-d1ff3e88262c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:30.920280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.920221    4877 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-routing-overlay-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:30.921970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.921937    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz" pod "b2e83bd0-4820-4f38-8f86-d1ff3e88262c" (UID: "b2e83bd0-4820-4f38-8f86-d1ff3e88262c"). InnerVolumeSpecName "kube-api-access-p9dgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:30.922096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:30.922025    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials" pod "b2e83bd0-4820-4f38-8f86-d1ff3e88262c" (UID: "b2e83bd0-4820-4f38-8f86-d1ff3e88262c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:31.021870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.021837    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9dgz\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-kube-api-access-p9dgz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:31.021870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.021867    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:31.021870 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.021877    4877 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e83bd0-4820-4f38-8f86-d1ff3e88262c-provider-credentials\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:31.670008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.669725    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"
Oct 30 03:47:31.670008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.669738    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz" event={"ID":"b2e83bd0-4820-4f38-8f86-d1ff3e88262c","Type":"ContainerDied","Data":"19a2d9e848412bc18aa1c32133469e9a4adcc5f18b40b6d7bbfdc768020c5edb"}
Oct 30 03:47:31.670008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.669822    4877 scope.go:122] "RemoveContainer" containerID="1d02c1b2d23e09632864c1caccce1caedcc8a40a0249ced7ec71b2da2ab714bb"
Oct 30 03:47:31.723168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.723132    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"]
Oct 30 03:47:31.735139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.735115    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b7wvfwz"]
Oct 30 03:47:31.750581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.750558    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"]
Oct 30 03:47:31.751307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751286    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5f315c41-cf78-483e-9d57-2237b2d9ee12" containerName="revoke-keys"
Oct 30 03:47:31.751307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751305    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f315c41-cf78-483e-9d57-2237b2d9ee12" containerName="revoke-keys"
Oct 30 03:47:31.751490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751340    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b2e83bd0-4820-4f38-8f86-d1ff3e88262c" containerName="payload-processing"
Oct 30 03:47:31.751490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751346    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e83bd0-4820-4f38-8f86-d1ff3e88262c" containerName="payload-processing"
Oct 30 03:47:31.751490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751365    4877 cpu_manager.go:414] "containerMap: removing container" podUID="46f6d363-edf0-477b-b114-6d80b7d192b5" containerName="revoke-keys"
Oct 30 03:47:31.751490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751370    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f6d363-edf0-477b-b114-6d80b7d192b5" containerName="revoke-keys"
Oct 30 03:47:31.751619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751514    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e83bd0-4820-4f38-8f86-d1ff3e88262c" containerName="payload-processing"
Oct 30 03:47:31.751619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751526    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f6d363-edf0-477b-b114-6d80b7d192b5" containerName="revoke-keys"
Oct 30 03:47:31.751619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.751538    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f315c41-cf78-483e-9d57-2237b2d9ee12" containerName="revoke-keys"
Oct 30 03:47:31.756741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.756724    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.771482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.771458    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:47:31.771829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.771626    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:47:31.775646 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.775623    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"]
Oct 30 03:47:31.834787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.834753    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.834956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.834795    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9csm\" (UniqueName: \"kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.834956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.834888    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.940704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.936633    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.940704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.937095    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.940704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.937133    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9csm\" (UniqueName: \"kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.940704 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.937804    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.958488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.950890    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:31.959866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:31.959822    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9csm\" (UniqueName: \"kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm\") pod \"maas-api-revoke-sub-b457f708-chdkw\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:32.072335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:32.072301    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:32.382743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:32.382713    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"]
Oct 30 03:47:32.385749 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:47:32.385722    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f50ae9_00d4_450a_a8c7_97e6463378d7.slice/crio-5af76177f497db3355c5d0a2b504a5799a1fb3fc2365c97e630be2ad244d3b51 WatchSource:0}: Error finding container 5af76177f497db3355c5d0a2b504a5799a1fb3fc2365c97e630be2ad244d3b51: Status 404 returned error can't find the container with id 5af76177f497db3355c5d0a2b504a5799a1fb3fc2365c97e630be2ad244d3b51
Oct 30 03:47:32.689814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:32.689782    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw" event={"ID":"16f50ae9-00d4-450a-a8c7-97e6463378d7","Type":"ContainerStarted","Data":"5af76177f497db3355c5d0a2b504a5799a1fb3fc2365c97e630be2ad244d3b51"}
Oct 30 03:47:33.745262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:33.745166    4877 generic.go:356] "Generic (PLEG): container finished" podID="16f50ae9-00d4-450a-a8c7-97e6463378d7" containerID="a9f337bb65ad7a3a1f5dcbf10d799fe6c04f515b88ec55ba9d86a566b4abec64" exitCode=0
Oct 30 03:47:33.751809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:33.751681    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e83bd0-4820-4f38-8f86-d1ff3e88262c" path="/var/lib/kubelet/pods/b2e83bd0-4820-4f38-8f86-d1ff3e88262c/volumes"
Oct 30 03:47:33.752095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:33.752071    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw" event={"ID":"16f50ae9-00d4-450a-a8c7-97e6463378d7","Type":"ContainerDied","Data":"a9f337bb65ad7a3a1f5dcbf10d799fe6c04f515b88ec55ba9d86a566b4abec64"}
Oct 30 03:47:35.182451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.178214    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:35.266448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.260585    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm\" (UniqueName: \"kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm\") pod \"16f50ae9-00d4-450a-a8c7-97e6463378d7\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") "
Oct 30 03:47:35.266448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.260719    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca\") pod \"16f50ae9-00d4-450a-a8c7-97e6463378d7\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") "
Oct 30 03:47:35.266448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.260812    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca\") pod \"16f50ae9-00d4-450a-a8c7-97e6463378d7\" (UID: \"16f50ae9-00d4-450a-a8c7-97e6463378d7\") "
Oct 30 03:47:35.270450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.268732    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca" pod "16f50ae9-00d4-450a-a8c7-97e6463378d7" (UID: "16f50ae9-00d4-450a-a8c7-97e6463378d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:47:35.271467 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.271370    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca" pod "16f50ae9-00d4-450a-a8c7-97e6463378d7" (UID: "16f50ae9-00d4-450a-a8c7-97e6463378d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:47:35.281448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.278571    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm" pod "16f50ae9-00d4-450a-a8c7-97e6463378d7" (UID: "16f50ae9-00d4-450a-a8c7-97e6463378d7"). InnerVolumeSpecName "kube-api-access-j9csm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:47:35.370505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.370177    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9csm\" (UniqueName: \"kubernetes.io/projected/16f50ae9-00d4-450a-a8c7-97e6463378d7-kube-api-access-j9csm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.370505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.370212    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.370505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.370230    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f50ae9-00d4-450a-a8c7-97e6463378d7-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:47:35.779655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.779620    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"
Oct 30 03:47:35.779828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.779644    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw" event={"ID":"16f50ae9-00d4-450a-a8c7-97e6463378d7","Type":"ContainerDied","Data":"5af76177f497db3355c5d0a2b504a5799a1fb3fc2365c97e630be2ad244d3b51"}
Oct 30 03:47:35.779828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:35.779702    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af76177f497db3355c5d0a2b504a5799a1fb3fc2365c97e630be2ad244d3b51"
Oct 30 03:47:46.231627 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:47:46.231581    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:47:48.029433 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:48.029393    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"]
Oct 30 03:47:48.041041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:48.041011    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56426188-wrtz7"]
Oct 30 03:47:49.304473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.304433    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"]
Oct 30 03:47:49.325335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.325300    4877 cpu_manager.go:414] "containerMap: removing container" podUID="16f50ae9-00d4-450a-a8c7-97e6463378d7" containerName="revoke-keys"
Oct 30 03:47:49.325590 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.325569    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f50ae9-00d4-450a-a8c7-97e6463378d7" containerName="revoke-keys"
Oct 30 03:47:49.326405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.326380    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f50ae9-00d4-450a-a8c7-97e6463378d7" containerName="revoke-keys"
Oct 30 03:47:49.382250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.366809    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.438801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.438753    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ae0e75-openshift-default-dockercfg-56mxg\""
Oct 30 03:47:49.498876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.498828    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.690634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.690542    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.715555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.715518    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.738543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.672609    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"]
Oct 30 03:47:49.743098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.738570    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.743098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.741722    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.743098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.741799    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.743098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.741878    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.743098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.741973    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.743098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.742039    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zlz\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.769806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.769771    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716aac94-a59f-44fb-a2f2-c4af7ac1680d" path="/var/lib/kubelet/pods/716aac94-a59f-44fb-a2f2-c4af7ac1680d/volumes"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.842953    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843009    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zlz\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843040    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843073    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843092    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843139    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843219    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843255    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.843586 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843303    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.844306 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.843906    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.844306 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.844176    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.844481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.844417    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.844545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.844497    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.844838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.844807    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.855465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.854770    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.860031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.860006    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.872285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.872217    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zlz\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:49.873823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:49.873797    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token\") pod \"e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") " pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:50.033702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.033669    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:50.034080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.033899    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"]
Oct 30 03:47:50.042582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.041021    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb61edbd-4598x"]
Oct 30 03:47:50.412453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.408087    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"]
Oct 30 03:47:50.432293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.432241    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:50.432624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.432593    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:50.433135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:50.433104    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:51.296848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:51.296812    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" event={"ID":"0f1620d8-851c-4b2e-bacf-be013a1cb957","Type":"ContainerStarted","Data":"b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"}
Oct 30 03:47:51.297181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:51.297140    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" event={"ID":"0f1620d8-851c-4b2e-bacf-be013a1cb957","Type":"ContainerStarted","Data":"9c1c0b460a563fb6a036b9adeb859f9c1f164f19faa59538dc421b9eddba17cb"}
Oct 30 03:47:51.793497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:51.778826    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f272c3a-3525-407a-9f51-90cd882b79b9" path="/var/lib/kubelet/pods/2f272c3a-3525-407a-9f51-90cd882b79b9/volumes"
Oct 30 03:47:52.040197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.040166    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:52.056258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.056181    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:52.156806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.156733    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podStartSLOduration=3.156712838 podStartE2EDuration="3.156712838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:51.360709406 +0000 UTC m=+2699.789598849" watchObservedRunningTime="2026-10-30 03:47:52.156712838 +0000 UTC m=+2700.585602256"
Oct 30 03:47:52.363499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.355222    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:52.465616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.465231    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:47:52.730790 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.730711    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:47:52.740121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:52.740086    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:47:58.865348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.865314    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"]
Oct 30 03:47:58.877075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.877042    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.881758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.881723    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.883536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.883508    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.883692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.883669    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgkg\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.884193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.884169    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.884385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.884367    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.884919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.884892    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.885045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.884963    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.885117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.885040    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.885117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.885105    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.901284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.901263    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d53363-openshift-default-dockercfg-bh4js\""
Oct 30 03:47:58.901508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.901358    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"]
Oct 30 03:47:58.986740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.986706    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.986912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.986786    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.986912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.986819    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.986959    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987030    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987123    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987187 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987161    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987194    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgkg\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987267    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987287    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987315    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987648    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987707    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.987743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.987714    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.989613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.989585    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:58.990257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:58.990235    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:59.005713 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.005681    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:59.005862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.005842    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgkg\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:59.204509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.200792    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:47:59.722075 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:47:59.722037    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68516de_5dcd_4571_8208_937bddb2108c.slice/crio-7e340e741621901de6f5fd28a468d9c2a55f81de92ce3c34645adab96f04c01f WatchSource:0}: Error finding container 7e340e741621901de6f5fd28a468d9c2a55f81de92ce3c34645adab96f04c01f: Status 404 returned error can't find the container with id 7e340e741621901de6f5fd28a468d9c2a55f81de92ce3c34645adab96f04c01f
Oct 30 03:47:59.725133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.725096    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:59.725248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.725178    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:59.725248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.725239    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:47:59.726088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:47:59.726067    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"]
Oct 30 03:48:00.452347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:00.452308    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" event={"ID":"f68516de-5dcd-4571-8208-937bddb2108c","Type":"ContainerStarted","Data":"14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"}
Oct 30 03:48:00.452765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:00.452361    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" event={"ID":"f68516de-5dcd-4571-8208-937bddb2108c","Type":"ContainerStarted","Data":"7e340e741621901de6f5fd28a468d9c2a55f81de92ce3c34645adab96f04c01f"}
Oct 30 03:48:00.514849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:00.514798    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podStartSLOduration=2.514783443 podStartE2EDuration="2.514783443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:00.513512209 +0000 UTC m=+2708.942401646" watchObservedRunningTime="2026-10-30 03:48:00.514783443 +0000 UTC m=+2708.943672936"
Oct 30 03:48:01.201766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:01.201731    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:48:01.208025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:01.207995    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:48:01.461650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:01.461573    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:48:01.462643 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:01.462618    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:48:14.876318 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:14.876282    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"]
Oct 30 03:48:14.876944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:14.876583    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy" containerID="cri-o://ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d" gracePeriod=30
Oct 30 03:48:16.238621 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:48:16.238573    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:48:20.322972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.322944    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:48:20.382041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382000    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382224    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382278    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382338    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382372    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382406    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382466    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382495    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.382690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.382658    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket\") pod \"182b3d24-bb3c-41d1-bd43-acccb34ac751\" (UID: \"182b3d24-bb3c-41d1-bd43-acccb34ac751\") "
Oct 30 03:48:20.383691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.383660    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:20.383895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.383819    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:20.384079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.384054    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:20.384154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.384073    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:20.384213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.384167    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:20.387805 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.387778    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "kube-api-access-rw584". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:20.387937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.387812    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:48:20.387937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.387866    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:20.389186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.389158    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy" pod "182b3d24-bb3c-41d1-bd43-acccb34ac751" (UID: "182b3d24-bb3c-41d1-bd43-acccb34ac751"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501539    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501587    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501610    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501631    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501657    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw584\" (UniqueName: \"kubernetes.io/projected/182b3d24-bb3c-41d1-bd43-acccb34ac751-kube-api-access-rw584\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501679    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501698    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501717    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182b3d24-bb3c-41d1-bd43-acccb34ac751-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.504085 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.501733    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182b3d24-bb3c-41d1-bd43-acccb34ac751-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:48:20.723686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.723598    4877 generic.go:356] "Generic (PLEG): container finished" podID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerID="ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d" exitCode=0
Oct 30 03:48:20.723686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.723665    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"
Oct 30 03:48:20.723899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.723684    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" event={"ID":"182b3d24-bb3c-41d1-bd43-acccb34ac751","Type":"ContainerDied","Data":"ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d"}
Oct 30 03:48:20.723899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.723738    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6" event={"ID":"182b3d24-bb3c-41d1-bd43-acccb34ac751","Type":"ContainerDied","Data":"b132738cf6db6549290e35a3a57dbfc786687cdeeac73fac94f903212bb73033"}
Oct 30 03:48:20.723899 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.723767    4877 scope.go:122] "RemoveContainer" containerID="ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d"
Oct 30 03:48:20.736358 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.736341    4877 scope.go:122] "RemoveContainer" containerID="ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d"
Oct 30 03:48:20.736671 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:48:20.736650    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d\": container with ID starting with ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d not found: ID does not exist" containerID="ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d"
Oct 30 03:48:20.736748 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.736683    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d"} err="failed to get container status \"ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d\": rpc error: code = NotFound desc = could not find container \"ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d\": container with ID starting with ecce6255f3e2e62ef892bbca93b8d5e98ebd7c12f73b76758c9988711f53249d not found: ID does not exist"
Oct 30 03:48:20.776072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.776043    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"]
Oct 30 03:48:20.780841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:20.780818    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-pdjp6"]
Oct 30 03:48:21.699539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:21.699507    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" path="/var/lib/kubelet/pods/182b3d24-bb3c-41d1-bd43-acccb34ac751/volumes"
Oct 30 03:48:46.230365 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:48:46.230285    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:48:55.451958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:55.451924    4877 scope.go:122] "RemoveContainer" containerID="9458a057611fbb69607c645cd1693dadbf54f40c68389a40655b09389bf32c30"
Oct 30 03:48:55.462953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:48:55.462926    4877 scope.go:122] "RemoveContainer" containerID="f9c1efbb8534661d662ccbf8fcbacd261a08de30d8d2ac6db82a47f59ba7470d"
Oct 30 03:49:02.710516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.710478    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"]
Oct 30 03:49:02.711348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.711328    4877 cpu_manager.go:414] "containerMap: removing container" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy"
Oct 30 03:49:02.711447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.711350    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy"
Oct 30 03:49:02.711505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.711495    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="182b3d24-bb3c-41d1-bd43-acccb34ac751" containerName="istio-proxy"
Oct 30 03:49:02.716472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.716452    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.743610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.743579    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:49:02.743610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.743577    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:49:02.749319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.749287    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"]
Oct 30 03:49:02.771325 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.771290    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7946z\" (UniqueName: \"kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.771483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.771371    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.771554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.771538    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.873456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.873395    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7946z\" (UniqueName: \"kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.873640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.873496    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.873640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.873608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.874294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.874274    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.876076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.876050    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:02.892847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:02.892821    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7946z\" (UniqueName: \"kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z\") pod \"maas-api-revoke-sub-e4725d63-xbchw\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:03.031407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:03.031380    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:03.391679 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:49:03.391642    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf601240_2c67_4ab7_8a64_17d6f390dabb.slice/crio-d63b41171971046b82b0246c63ea72ab348bd921fa2f6993ae67c5f7e8c75fdd WatchSource:0}: Error finding container d63b41171971046b82b0246c63ea72ab348bd921fa2f6993ae67c5f7e8c75fdd: Status 404 returned error can't find the container with id d63b41171971046b82b0246c63ea72ab348bd921fa2f6993ae67c5f7e8c75fdd
Oct 30 03:49:03.394199 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:03.394174    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"]
Oct 30 03:49:04.258913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:04.258815    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw" event={"ID":"af601240-2c67-4ab7-8a64-17d6f390dabb","Type":"ContainerStarted","Data":"09ae3ab46c64205aa961c61ad576abf5288331b2994029d794f4d44252f4c0d8"}
Oct 30 03:49:04.258913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:04.258875    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw" event={"ID":"af601240-2c67-4ab7-8a64-17d6f390dabb","Type":"ContainerStarted","Data":"d63b41171971046b82b0246c63ea72ab348bd921fa2f6993ae67c5f7e8c75fdd"}
Oct 30 03:49:04.301655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:04.301603    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw" podStartSLOduration=2.301589042 podStartE2EDuration="2.301589042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:04.296940363 +0000 UTC m=+2772.725829806" watchObservedRunningTime="2026-10-30 03:49:04.301589042 +0000 UTC m=+2772.730478459"
Oct 30 03:49:05.271775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:05.271732    4877 generic.go:356] "Generic (PLEG): container finished" podID="af601240-2c67-4ab7-8a64-17d6f390dabb" containerID="09ae3ab46c64205aa961c61ad576abf5288331b2994029d794f4d44252f4c0d8" exitCode=0
Oct 30 03:49:05.272222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:05.271773    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw" event={"ID":"af601240-2c67-4ab7-8a64-17d6f390dabb","Type":"ContainerDied","Data":"09ae3ab46c64205aa961c61ad576abf5288331b2994029d794f4d44252f4c0d8"}
Oct 30 03:49:06.507335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.507306    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:06.560638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.560602    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca\") pod \"af601240-2c67-4ab7-8a64-17d6f390dabb\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") "
Oct 30 03:49:06.560823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.560692    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z\" (UniqueName: \"kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z\") pod \"af601240-2c67-4ab7-8a64-17d6f390dabb\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") "
Oct 30 03:49:06.560823 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.560818    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca\") pod \"af601240-2c67-4ab7-8a64-17d6f390dabb\" (UID: \"af601240-2c67-4ab7-8a64-17d6f390dabb\") "
Oct 30 03:49:06.561040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.561014    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca" pod "af601240-2c67-4ab7-8a64-17d6f390dabb" (UID: "af601240-2c67-4ab7-8a64-17d6f390dabb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:06.561475 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.561451    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:49:06.563342 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.563306    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z" pod "af601240-2c67-4ab7-8a64-17d6f390dabb" (UID: "af601240-2c67-4ab7-8a64-17d6f390dabb"). InnerVolumeSpecName "kube-api-access-7946z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:06.563917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.563874    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca" pod "af601240-2c67-4ab7-8a64-17d6f390dabb" (UID: "af601240-2c67-4ab7-8a64-17d6f390dabb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:06.663714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.663672    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7946z\" (UniqueName: \"kubernetes.io/projected/af601240-2c67-4ab7-8a64-17d6f390dabb-kube-api-access-7946z\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:49:06.663714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:06.663708    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af601240-2c67-4ab7-8a64-17d6f390dabb-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:49:07.326192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:07.326161    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"
Oct 30 03:49:07.326192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:07.326180    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw" event={"ID":"af601240-2c67-4ab7-8a64-17d6f390dabb","Type":"ContainerDied","Data":"d63b41171971046b82b0246c63ea72ab348bd921fa2f6993ae67c5f7e8c75fdd"}
Oct 30 03:49:07.326489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:07.326230    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d63b41171971046b82b0246c63ea72ab348bd921fa2f6993ae67c5f7e8c75fdd"
Oct 30 03:49:13.333528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.333493    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"]
Oct 30 03:49:13.334256 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.334239    4877 cpu_manager.go:414] "containerMap: removing container" podUID="af601240-2c67-4ab7-8a64-17d6f390dabb" containerName="revoke-keys"
Oct 30 03:49:13.334303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.334257    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af601240-2c67-4ab7-8a64-17d6f390dabb" containerName="revoke-keys"
Oct 30 03:49:13.334406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.334396    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="af601240-2c67-4ab7-8a64-17d6f390dabb" containerName="revoke-keys"
Oct 30 03:49:13.338291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.338268    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.358096 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.358069    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"]
Oct 30 03:49:13.376696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.376666    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:49:13.376986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.376956    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:49:13.473912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.473876    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.474090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.474066    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhtp\" (UniqueName: \"kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.474142 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.474123    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.575195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.575155    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhtp\" (UniqueName: \"kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.575409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.575241    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.575523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.575413    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.576113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.576082    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.578128 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.578104    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.595197 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.595138    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhtp\" (UniqueName: \"kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp\") pod \"maas-api-revoke-sub-37628bc8-zpb7d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:13.653828 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:13.653797    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:14.164320 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:49:14.164278    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29acc96_fa42_4214_bbb8_9e33adc52a2d.slice/crio-89bdf2b7aac04674d482fcfed5441e076c8841cc965bb80374a1cc85348d7cef WatchSource:0}: Error finding container 89bdf2b7aac04674d482fcfed5441e076c8841cc965bb80374a1cc85348d7cef: Status 404 returned error can't find the container with id 89bdf2b7aac04674d482fcfed5441e076c8841cc965bb80374a1cc85348d7cef
Oct 30 03:49:14.167701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:14.167668    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"]
Oct 30 03:49:14.422470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:14.421840    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d" event={"ID":"e29acc96-fa42-4214-bbb8-9e33adc52a2d","Type":"ContainerStarted","Data":"89bdf2b7aac04674d482fcfed5441e076c8841cc965bb80374a1cc85348d7cef"}
Oct 30 03:49:15.448028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:15.447696    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d" event={"ID":"e29acc96-fa42-4214-bbb8-9e33adc52a2d","Type":"ContainerStarted","Data":"8630c9aad7f264e6929a9c7f795aae13b02ecd00aab4ccbb330ca42f44b71141"}
Oct 30 03:49:16.331243 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:49:16.331191    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:49:16.467253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.465180    4877 generic.go:356] "Generic (PLEG): container finished" podID="e29acc96-fa42-4214-bbb8-9e33adc52a2d" containerID="8630c9aad7f264e6929a9c7f795aae13b02ecd00aab4ccbb330ca42f44b71141" exitCode=0
Oct 30 03:49:16.467253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.465263    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d" event={"ID":"e29acc96-fa42-4214-bbb8-9e33adc52a2d","Type":"ContainerDied","Data":"8630c9aad7f264e6929a9c7f795aae13b02ecd00aab4ccbb330ca42f44b71141"}
Oct 30 03:49:16.528597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.527458    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"]
Oct 30 03:49:16.534552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.534064    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.544835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.544735    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"]
Oct 30 03:49:16.594121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.594023    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e84013-openshift-default-dockercfg-4sn84\""
Oct 30 03:49:16.692534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.668093    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.692972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.692917    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.708541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.708398    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqsz7\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.709465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.709260    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.709680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.709659    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.715869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.715845    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.723772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.723750    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.744406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.744212    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.744618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.744599    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.876957    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877036    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877076    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877243    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877287    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877314    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqsz7\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877409    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877454    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.899595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.877546    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.936399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.936361    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.936958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.936926    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.937721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.937697    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.938447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.938407    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:16.964758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:16.964714    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:17.039691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.015404    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqsz7\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:17.039691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.024330    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:17.039691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.025277    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:17.039691 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.026492    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:17.147950 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.147921    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:17.210959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.210885    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp\" (UniqueName: \"kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp\") pod \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") "
Oct 30 03:49:17.211172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.211149    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca\") pod \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") "
Oct 30 03:49:17.211231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.211214    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca\") pod \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\" (UID: \"e29acc96-fa42-4214-bbb8-9e33adc52a2d\") "
Oct 30 03:49:17.211572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.211547    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca" pod "e29acc96-fa42-4214-bbb8-9e33adc52a2d" (UID: "e29acc96-fa42-4214-bbb8-9e33adc52a2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:49:17.211800 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.211784    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:49:17.213121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.213096    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca" pod "e29acc96-fa42-4214-bbb8-9e33adc52a2d" (UID: "e29acc96-fa42-4214-bbb8-9e33adc52a2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:49:17.213224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.213164    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp" pod "e29acc96-fa42-4214-bbb8-9e33adc52a2d" (UID: "e29acc96-fa42-4214-bbb8-9e33adc52a2d"). InnerVolumeSpecName "kube-api-access-gzhtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:49:17.234332 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.234309    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:17.313734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.313703    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29acc96-fa42-4214-bbb8-9e33adc52a2d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:49:17.313734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.313731    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzhtp\" (UniqueName: \"kubernetes.io/projected/e29acc96-fa42-4214-bbb8-9e33adc52a2d-kube-api-access-gzhtp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:49:17.508505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.503525    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d" event={"ID":"e29acc96-fa42-4214-bbb8-9e33adc52a2d","Type":"ContainerDied","Data":"89bdf2b7aac04674d482fcfed5441e076c8841cc965bb80374a1cc85348d7cef"}
Oct 30 03:49:17.508505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.503600    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89bdf2b7aac04674d482fcfed5441e076c8841cc965bb80374a1cc85348d7cef"
Oct 30 03:49:17.508505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.503677    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"
Oct 30 03:49:17.606784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.606750    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"]
Oct 30 03:49:17.608018 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:49:17.607991    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46386d6_80fd_40a9_aff1_1db047cfb8b9.slice/crio-1971a272ebd5294a90b0443257e921090308407f901ca55d120130f09fcb46df WatchSource:0}: Error finding container 1971a272ebd5294a90b0443257e921090308407f901ca55d120130f09fcb46df: Status 404 returned error can't find the container with id 1971a272ebd5294a90b0443257e921090308407f901ca55d120130f09fcb46df
Oct 30 03:49:17.611117 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.611083    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:17.611206 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.611157    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:17.611247 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:17.611202    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:18.521721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:18.521667    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" event={"ID":"e46386d6-80fd-40a9-aff1-1db047cfb8b9","Type":"ContainerStarted","Data":"06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"}
Oct 30 03:49:18.522190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:18.521747    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" event={"ID":"e46386d6-80fd-40a9-aff1-1db047cfb8b9","Type":"ContainerStarted","Data":"1971a272ebd5294a90b0443257e921090308407f901ca55d120130f09fcb46df"}
Oct 30 03:49:18.609133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:18.603396    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podStartSLOduration=2.603373711 podStartE2EDuration="2.603373711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:18.574565224 +0000 UTC m=+2787.003454638" watchObservedRunningTime="2026-10-30 03:49:18.603373711 +0000 UTC m=+2787.032263131"
Oct 30 03:49:19.234765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:19.234723    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:19.240911 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:19.240883    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:19.539146 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:19.539114    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:19.540328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:19.540306    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:49:22.143204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.143168    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"]
Oct 30 03:49:22.143989 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.143971    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e29acc96-fa42-4214-bbb8-9e33adc52a2d" containerName="revoke-keys"
Oct 30 03:49:22.144074 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.143990    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29acc96-fa42-4214-bbb8-9e33adc52a2d" containerName="revoke-keys"
Oct 30 03:49:22.144185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.144172    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29acc96-fa42-4214-bbb8-9e33adc52a2d" containerName="revoke-keys"
Oct 30 03:49:22.147995 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.147969    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.167866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.167838    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"]
Oct 30 03:49:22.172840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.172816    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"openshift-service-ca.crt\""
Oct 30 03:49:22.173048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.172823    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"routing-overlay\""
Oct 30 03:49:22.173169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.172899    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"kube-root-ca.crt\""
Oct 30 03:49:22.173169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.172931    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689bcf\""
Oct 30 03:49:22.251019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.250983    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"agc-cred-167b6dd59f53\""
Oct 30 03:49:22.251019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.251024    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"agc-cred-7b4014be3195\""
Oct 30 03:49:22.251230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.251056    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-e2e--22d06fb9-dockercfg-kmzx9\""
Oct 30 03:49:22.271347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.271312    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.271555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.271495    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.271676 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.271651    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.271769 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.271755    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r45wg\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.373737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.373696    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.373954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.373793    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.373954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.373881    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.373954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.373947    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r45wg\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.374943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.374912    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.375084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.374991    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.376737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.376712    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.399137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.399081    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r45wg\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.465613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.465568    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:22.806149 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:49:22.806116    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739f1aec_7e90_4787_80e8_18d13f43849e.slice/crio-02f49d4487a48c64fa2bde2fb568d469819bb54d4a630c0419cf104af12301be WatchSource:0}: Error finding container 02f49d4487a48c64fa2bde2fb568d469819bb54d4a630c0419cf104af12301be: Status 404 returned error can't find the container with id 02f49d4487a48c64fa2bde2fb568d469819bb54d4a630c0419cf104af12301be
Oct 30 03:49:22.811500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:22.811476    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"]
Oct 30 03:49:23.269974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.269889    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"]
Oct 30 03:49:23.278710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.278687    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.293749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.293717    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e84013-dockercfg-np6rw\""
Oct 30 03:49:23.295837 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.295808    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"]
Oct 30 03:49:23.300762 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.300724    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e84013\""
Oct 30 03:49:23.563835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.563511    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:49:23.563835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.563573    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:49:23.597446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.581637    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"]
Oct 30 03:49:23.604659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.601019    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.604659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.601380    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6tk\" (UniqueName: \"kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk\") pod \"payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.635446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.619468    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"]
Oct 30 03:49:23.635446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.619596    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:23.675944 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.675901    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w" event={"ID":"739f1aec-7e90-4787-80e8-18d13f43849e","Type":"ContainerStarted","Data":"baddcc94b39bdcd3d32ef1f511a1000c7d61294b1c767a5fc6697bb83bd6e0a3"}
Oct 30 03:49:23.676183 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.675960    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w" event={"ID":"739f1aec-7e90-4787-80e8-18d13f43849e","Type":"ContainerStarted","Data":"02f49d4487a48c64fa2bde2fb568d469819bb54d4a630c0419cf104af12301be"}
Oct 30 03:49:23.676579 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.676540    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:23.703736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.703694    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6tk\" (UniqueName: \"kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk\") pod \"payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.703902 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.703869    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.704930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.704902    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.725852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.725819    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6tk\" (UniqueName: \"kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk\") pod \"payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.736499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.736449    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w" podStartSLOduration=1.736433049 podStartE2EDuration="1.736433049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:23.732673211 +0000 UTC m=+2792.161562630" watchObservedRunningTime="2026-10-30 03:49:23.736433049 +0000 UTC m=+2792.165322459"
Oct 30 03:49:23.805839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.805793    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:23.806039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.805950    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46r2x\" (UniqueName: \"kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x\") pod \"payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:23.899628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.899543    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:23.907321 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.907283    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:23.907561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.907407    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46r2x\" (UniqueName: \"kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x\") pod \"payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:23.908070 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.908046    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:23.926880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:23.926860    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46r2x\" (UniqueName: \"kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x\") pod \"payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:24.027877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.027825    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:24.236866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.236835    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"]
Oct 30 03:49:24.240417 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:49:24.240229    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b99d5ad_c664_4fae_b8f8_ea76535a03c5.slice/crio-08197d81e27ba79925489091da700794cbfd1e3f7daa5dd33bac054235c75e80 WatchSource:0}: Error finding container 08197d81e27ba79925489091da700794cbfd1e3f7daa5dd33bac054235c75e80: Status 404 returned error can't find the container with id 08197d81e27ba79925489091da700794cbfd1e3f7daa5dd33bac054235c75e80
Oct 30 03:49:24.387818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.387781    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"]
Oct 30 03:49:24.401561 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:49:24.401526    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b372d9_e957_4c98_ad09_a114e9c0e387.slice/crio-81f7fac442e71c696f38b8891feead811e643333c95c30652980b6f303672ae9 WatchSource:0}: Error finding container 81f7fac442e71c696f38b8891feead811e643333c95c30652980b6f303672ae9: Status 404 returned error can't find the container with id 81f7fac442e71c696f38b8891feead811e643333c95c30652980b6f303672ae9
Oct 30 03:49:24.794032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.793937    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" event={"ID":"74b372d9-e957-4c98-ad09-a114e9c0e387","Type":"ContainerStarted","Data":"81f7fac442e71c696f38b8891feead811e643333c95c30652980b6f303672ae9"}
Oct 30 03:49:24.797964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.797812    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" event={"ID":"9b99d5ad-c664-4fae-b8f8-ea76535a03c5","Type":"ContainerStarted","Data":"f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759"}
Oct 30 03:49:24.797964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.797863    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" event={"ID":"9b99d5ad-c664-4fae-b8f8-ea76535a03c5","Type":"ContainerStarted","Data":"08197d81e27ba79925489091da700794cbfd1e3f7daa5dd33bac054235c75e80"}
Oct 30 03:49:24.797964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.797907    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:24.912635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:24.912491    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" podStartSLOduration=1.912469825 podStartE2EDuration="1.912469825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:24.874713906 +0000 UTC m=+2793.303603325" watchObservedRunningTime="2026-10-30 03:49:24.912469825 +0000 UTC m=+2793.341359243"
Oct 30 03:49:25.830838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:25.830350    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" event={"ID":"74b372d9-e957-4c98-ad09-a114e9c0e387","Type":"ContainerStarted","Data":"0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab"}
Oct 30 03:49:25.830838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:25.830617    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:25.886153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:25.886090    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" podStartSLOduration=2.886070212 podStartE2EDuration="2.886070212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:25.869628033 +0000 UTC m=+2794.298517482" watchObservedRunningTime="2026-10-30 03:49:25.886070212 +0000 UTC m=+2794.314959635"
Oct 30 03:49:26.848806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:26.848774    4877 generic.go:356] "Generic (PLEG): container finished" podID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerID="b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710" exitCode=0
Oct 30 03:49:26.849238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:26.848850    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" event={"ID":"0f1620d8-851c-4b2e-bacf-be013a1cb957","Type":"ContainerDied","Data":"b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"}
Oct 30 03:49:26.850110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:26.850089    4877 scope.go:122] "RemoveContainer" containerID="b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"
Oct 30 03:49:26.852308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:26.852275    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:26.852406 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:26.852389    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:26.852478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:26.852458    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:27.040660 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:27.040509    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"]
Oct 30 03:49:27.044139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:27.044109    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-021e6dfc-jvb6b"]
Oct 30 03:49:27.718345 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:27.718307    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84813703-5b47-472b-b87c-6eb3045bdcab" path="/var/lib/kubelet/pods/84813703-5b47-472b-b87c-6eb3045bdcab/volumes"
Oct 30 03:49:27.860263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:27.860231    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" event={"ID":"0f1620d8-851c-4b2e-bacf-be013a1cb957","Type":"ContainerStarted","Data":"71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0"}
Oct 30 03:49:28.039917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:28.039883    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:49:28.040291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:28.040264    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:28.040501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:28.040472    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:29.037474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:29.037010    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:29.037474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:29.037079    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:30.034999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.034957    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:30.035222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.035013    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:30.767315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.767272    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"]
Oct 30 03:49:30.773370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.773339    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:30.794713 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.794682    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-35fc87-openshift-default-dockercfg-n4t6x\""
Oct 30 03:49:30.859038 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.859002    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"]
Oct 30 03:49:30.932806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.891039    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:30.974515 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.933486    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:30.995588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.995541    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:30.995958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.995937    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bxm\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.017503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:30.996166    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.042451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.024891    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.042451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.033580    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.042451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.033714    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.054479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.046616    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.085454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.085089    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:31.085454 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.085149    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.158622    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.158700    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.158790    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bxm\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.158875    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.158904    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.158956    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.159029    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.159082    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.175528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.159268    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.177688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.182833    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.185468    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.186083    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.186300    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.186913    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.203455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.191016    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.239791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.227878    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bxm\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.266248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.266212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.465191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.465156    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:31.904694 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.904404    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"]
Oct 30 03:49:31.907601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.907566    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:31.907705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.907633    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:31.907705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:31.907661    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:32.034181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.034149    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:32.034299 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.034211    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:32.233485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.233309    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" event={"ID":"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b","Type":"ContainerStarted","Data":"f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"}
Oct 30 03:49:32.233485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.233370    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" event={"ID":"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b","Type":"ContainerStarted","Data":"40255224e3d0eada88b2cd983d8fc7ce1e3a47b6120204016eacc854d724ec59"}
Oct 30 03:49:32.282476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.278922    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podStartSLOduration=2.278902211 podStartE2EDuration="2.278902211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:32.27561696 +0000 UTC m=+2800.704506403" watchObservedRunningTime="2026-10-30 03:49:32.278902211 +0000 UTC m=+2800.707791632"
Oct 30 03:49:32.478453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.473056    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:32.520536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.517544    4877 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:49:32.520536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:32.517626    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:49:33.034236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:33.034198    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:33.034669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:33.034268    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:33.489635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:33.489556    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:34.035155 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:34.035114    4877 patch_prober.go:29] interesting pod/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 30 03:49:34.035690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:34.035193    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 30 03:49:34.275813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:34.275777    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:34.276909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:34.276880    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:49:35.040276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:35.040241    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:49:35.040825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:35.040679    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:49:35.041334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:35.041314    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:49:35.833869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:35.833839    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:49:36.890445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:36.888051    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:49:36.890445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:36.889576    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:49:37.312497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:37.312459    4877 generic.go:356] "Generic (PLEG): container finished" podID="f68516de-5dcd-4571-8208-937bddb2108c" containerID="14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e" exitCode=0
Oct 30 03:49:37.312695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:37.312488    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" event={"ID":"f68516de-5dcd-4571-8208-937bddb2108c","Type":"ContainerDied","Data":"14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"}
Oct 30 03:49:37.313099 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:37.313072    4877 scope.go:122] "RemoveContainer" containerID="14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"
Oct 30 03:49:37.315285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:37.315251    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:37.315399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:37.315323    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:37.315399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:37.315363    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:49:38.041552 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:38.041507    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"]
Oct 30 03:49:38.046131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:38.046099    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729bdfcf-g5qcm"]
Oct 30 03:49:38.358500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:38.355481    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" event={"ID":"f68516de-5dcd-4571-8208-937bddb2108c","Type":"ContainerStarted","Data":"b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82"}
Oct 30 03:49:39.028292 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.028251    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"]
Oct 30 03:49:39.037566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.037535    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-611f8e44-dqp77"]
Oct 30 03:49:39.203448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.203200    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:49:39.203448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.203365    4877 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:49:39.204046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.203498    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:49:39.712821 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.712784    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514cb366-8797-4b9e-82d7-c25ed094c45c" path="/var/lib/kubelet/pods/514cb366-8797-4b9e-82d7-c25ed094c45c/volumes"
Oct 30 03:49:39.713294 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:39.713277    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b" path="/var/lib/kubelet/pods/eb2e3226-0cd4-45a8-a7ff-ee904a0e8d4b/volumes"
Oct 30 03:49:40.208572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:40.204668    4877 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:49:40.208572 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:40.204726    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:49:41.204867 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:41.204827    4877 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:49:41.205179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:41.205148    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:49:42.204020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:42.203983    4877 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:49:42.204659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:42.204611    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:49:43.201580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:43.201537    4877 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 03:49:43.207559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:43.207513    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 03:49:44.206835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:44.206797    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:49:44.207254 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:44.207231    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:49:44.208012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:49:44.207985    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:49:46.230815 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:49:46.230773    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:50:13.926030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.925937    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"]
Oct 30 03:50:13.932250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.932224    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:13.951960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.951926    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"]
Oct 30 03:50:13.953749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.953725    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:50:13.954015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.953985    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:50:13.988336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.988306    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mgz\" (UniqueName: \"kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:13.988581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.988558    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:13.988708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:13.988690    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.089734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.089686    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.089954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.089825    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mgz\" (UniqueName: \"kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.090047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.090025    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.090889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.090860    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.092476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.092411    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.109468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.109440    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mgz\" (UniqueName: \"kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz\") pod \"maas-api-revoke-sub-37532b3a-4n7f4\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.248136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.248063    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:14.580048 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:50:14.580004    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ffe568_b300_4f22_ad1b_41fdf86d0fab.slice/crio-2d8ebcb7991c63a4965e23c9ee3c3c1ef19422fcbe617dfee57bd5b9a92db32f WatchSource:0}: Error finding container 2d8ebcb7991c63a4965e23c9ee3c3c1ef19422fcbe617dfee57bd5b9a92db32f: Status 404 returned error can't find the container with id 2d8ebcb7991c63a4965e23c9ee3c3c1ef19422fcbe617dfee57bd5b9a92db32f
Oct 30 03:50:14.580482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.580454    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"]
Oct 30 03:50:14.883394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:14.883301    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4" event={"ID":"b1ffe568-b300-4f22-ad1b-41fdf86d0fab","Type":"ContainerStarted","Data":"2d8ebcb7991c63a4965e23c9ee3c3c1ef19422fcbe617dfee57bd5b9a92db32f"}
Oct 30 03:50:15.893612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:15.893523    4877 generic.go:356] "Generic (PLEG): container finished" podID="b1ffe568-b300-4f22-ad1b-41fdf86d0fab" containerID="48192a98df0ad83c51f19cd776d52486e1d0a069b1efca914cec5ebba90b8694" exitCode=0
Oct 30 03:50:15.894065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:15.893617    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4" event={"ID":"b1ffe568-b300-4f22-ad1b-41fdf86d0fab","Type":"ContainerDied","Data":"48192a98df0ad83c51f19cd776d52486e1d0a069b1efca914cec5ebba90b8694"}
Oct 30 03:50:16.232018 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:50:16.231926    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:50:17.144373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.144348    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:17.254220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.254140    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz\" (UniqueName: \"kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz\") pod \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") "
Oct 30 03:50:17.254388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.254233    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca\") pod \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") "
Oct 30 03:50:17.254388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.254340    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca\") pod \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\" (UID: \"b1ffe568-b300-4f22-ad1b-41fdf86d0fab\") "
Oct 30 03:50:17.254698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.254669    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca" pod "b1ffe568-b300-4f22-ad1b-41fdf86d0fab" (UID: "b1ffe568-b300-4f22-ad1b-41fdf86d0fab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:17.255105 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.255086    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:50:17.256622 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.256588    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz" pod "b1ffe568-b300-4f22-ad1b-41fdf86d0fab" (UID: "b1ffe568-b300-4f22-ad1b-41fdf86d0fab"). InnerVolumeSpecName "kube-api-access-78mgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:17.256757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.256735    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca" pod "b1ffe568-b300-4f22-ad1b-41fdf86d0fab" (UID: "b1ffe568-b300-4f22-ad1b-41fdf86d0fab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:17.356409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.356373    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mgz\" (UniqueName: \"kubernetes.io/projected/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-kube-api-access-78mgz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:50:17.356409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.356407    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ffe568-b300-4f22-ad1b-41fdf86d0fab-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:50:17.955391 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.955359    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"
Oct 30 03:50:17.955635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.955397    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4" event={"ID":"b1ffe568-b300-4f22-ad1b-41fdf86d0fab","Type":"ContainerDied","Data":"2d8ebcb7991c63a4965e23c9ee3c3c1ef19422fcbe617dfee57bd5b9a92db32f"}
Oct 30 03:50:17.955635 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:17.955479    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8ebcb7991c63a4965e23c9ee3c3c1ef19422fcbe617dfee57bd5b9a92db32f"
Oct 30 03:50:27.645949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.645909    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"]
Oct 30 03:50:27.647215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.647183    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b1ffe568-b300-4f22-ad1b-41fdf86d0fab" containerName="revoke-keys"
Oct 30 03:50:27.647215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.647211    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ffe568-b300-4f22-ad1b-41fdf86d0fab" containerName="revoke-keys"
Oct 30 03:50:27.647554 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.647535    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ffe568-b300-4f22-ad1b-41fdf86d0fab" containerName="revoke-keys"
Oct 30 03:50:27.652343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.652320    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.667119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.667085    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3cfed7-openshift-default-dockercfg-52c28\""
Oct 30 03:50:27.683729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.683700    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"]
Oct 30 03:50:27.725048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725010    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725088    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsrd\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725206    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725283    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725432    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725532    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725555    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725603    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.725709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.725645    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.830570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.830522    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.830883 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.830863    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsrd\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.831042 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.831023    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.831186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.831170    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.840448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.831347    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.840803 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.840775    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.840942 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.840926    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.841095 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.841076    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.841241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.841226    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.858483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.842209    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.859581 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.859098    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.860054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.860029    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.864650 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.860305    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.877869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.877776    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.889455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.886020    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.889455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.886710    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.909937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.909863    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.929349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.929317    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsrd\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd\") pod \"e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") " pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:27.977008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:27.976717    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:28.420459 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:50:28.420399    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3660296_8703_4234_84cd_c8b96142c79f.slice/crio-7fecc357e08f55d8a72413bd93b4c30275db87c0a6729d87f0ac4dc6717fb3e8 WatchSource:0}: Error finding container 7fecc357e08f55d8a72413bd93b4c30275db87c0a6729d87f0ac4dc6717fb3e8: Status 404 returned error can't find the container with id 7fecc357e08f55d8a72413bd93b4c30275db87c0a6729d87f0ac4dc6717fb3e8
Oct 30 03:50:28.421219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:28.421195    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"]
Oct 30 03:50:28.423305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:28.423273    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:50:28.423388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:28.423337    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:50:28.423388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:28.423371    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:50:29.124897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:29.121182    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" event={"ID":"e3660296-8703-4234-84cd-c8b96142c79f","Type":"ContainerStarted","Data":"1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"}
Oct 30 03:50:29.124897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:29.121248    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" event={"ID":"e3660296-8703-4234-84cd-c8b96142c79f","Type":"ContainerStarted","Data":"7fecc357e08f55d8a72413bd93b4c30275db87c0a6729d87f0ac4dc6717fb3e8"}
Oct 30 03:50:29.185546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:29.185483    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podStartSLOduration=2.185464331 podStartE2EDuration="2.185464331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:29.171830735 +0000 UTC m=+2857.600720154" watchObservedRunningTime="2026-10-30 03:50:29.185464331 +0000 UTC m=+2857.614353750"
Oct 30 03:50:29.977499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:29.977468    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:29.983991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:29.983963    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:30.133266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:30.133233    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:30.134503 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:30.134476    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:50:33.097354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:33.097316    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"]
Oct 30 03:50:33.109153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:33.109124    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95911715-hrb44"]
Oct 30 03:50:33.924223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:33.924183    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3a6fc6-2044-48b6-935b-dcb778dd71bb" path="/var/lib/kubelet/pods/2e3a6fc6-2044-48b6-935b-dcb778dd71bb/volumes"
Oct 30 03:50:45.121448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:45.113471    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"]
Oct 30 03:50:45.142947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:45.142912    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98fb993-9dbz5"]
Oct 30 03:50:45.799106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:45.799070    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80426dbf-ea7e-436c-8b7d-9c35cb79a5be" path="/var/lib/kubelet/pods/80426dbf-ea7e-436c-8b7d-9c35cb79a5be/volumes"
Oct 30 03:50:46.367833 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:50:46.367797    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:50:49.551969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.551935    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"]
Oct 30 03:50:49.558372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.558350    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.599441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.599385    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:50:49.599834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.599812    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:50:49.647371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.647330    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"]
Oct 30 03:50:49.651807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.651769    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.652314 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.652291    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.652516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.652497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdfk\" (UniqueName: \"kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.753775 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.753741    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.754011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.753996    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdfk\" (UniqueName: \"kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.754516 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.754493    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.755172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.755145    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.758119 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.758095    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.774150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.774119    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdfk\" (UniqueName: \"kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk\") pod \"maas-api-revoke-sub-48d4ea2c-c4dbv\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:49.906739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:49.906646    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:50.570344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:50.569400    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"]
Oct 30 03:50:50.638912 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:50.638859    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv" event={"ID":"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd","Type":"ContainerStarted","Data":"bee31107fc700004a7efec7b5900cbcac78ee0e23e0ee48cba0b65a09ba043e7"}
Oct 30 03:50:51.655286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:51.655252    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv" event={"ID":"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd","Type":"ContainerStarted","Data":"18181811fcf4cd719fcac3f2680ede22791abef409244a5114e45cbb946ca673"}
Oct 30 03:50:51.700538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:51.700460    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv" podStartSLOduration=2.700417876 podStartE2EDuration="2.700417876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:51.690676637 +0000 UTC m=+2880.119566056" watchObservedRunningTime="2026-10-30 03:50:51.700417876 +0000 UTC m=+2880.129307295"
Oct 30 03:50:52.668725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:52.668689    4877 generic.go:356] "Generic (PLEG): container finished" podID="d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" containerID="18181811fcf4cd719fcac3f2680ede22791abef409244a5114e45cbb946ca673" exitCode=0
Oct 30 03:50:52.669156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:52.668791    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv" event={"ID":"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd","Type":"ContainerDied","Data":"18181811fcf4cd719fcac3f2680ede22791abef409244a5114e45cbb946ca673"}
Oct 30 03:50:54.046216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.046177    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:54.143553 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.143521    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca\") pod \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") "
Oct 30 03:50:54.143739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.143579    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca\") pod \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") "
Oct 30 03:50:54.143739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.143694    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk\" (UniqueName: \"kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk\") pod \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\" (UID: \"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd\") "
Oct 30 03:50:54.144048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.143992    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca" pod "d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" (UID: "d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:54.144909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.144885    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:50:54.162152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.162123    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk" pod "d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" (UID: "d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd"). InnerVolumeSpecName "kube-api-access-lxdfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:54.162152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.162123    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca" pod "d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" (UID: "d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:54.246246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.246166    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdfk\" (UniqueName: \"kubernetes.io/projected/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-kube-api-access-lxdfk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:50:54.246246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.246197    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:50:54.706705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.706673    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"
Oct 30 03:50:54.706974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.706702    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv" event={"ID":"d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd","Type":"ContainerDied","Data":"bee31107fc700004a7efec7b5900cbcac78ee0e23e0ee48cba0b65a09ba043e7"}
Oct 30 03:50:54.707121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:54.706995    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bee31107fc700004a7efec7b5900cbcac78ee0e23e0ee48cba0b65a09ba043e7"
Oct 30 03:50:54.764495 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:50:54.764413    4877 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bdb2d1_b9e3_49fb_9629_5a8f835f75dd.slice/crio-bee31107fc700004a7efec7b5900cbcac78ee0e23e0ee48cba0b65a09ba043e7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bdb2d1_b9e3_49fb_9629_5a8f835f75dd.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 03:50:55.522213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:55.522160    4877 scope.go:122] "RemoveContainer" containerID="1e86162a951a6ea06d22a7ece3a23257ef4236283bdfdcbea826f72deb8dfd55"
Oct 30 03:50:55.535620 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:55.535595    4877 scope.go:122] "RemoveContainer" containerID="0273499e3967a6c10e5ce39084721d44dded6b97c94834029b0210b32f083ac9"
Oct 30 03:50:55.545746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:50:55.545718    4877 scope.go:122] "RemoveContainer" containerID="78cca1a9e4e362243be15b98b95878eb8dd807f18ee121eda8f7ee5ad87f7694"
Oct 30 03:51:00.474125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.474079    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"]
Oct 30 03:51:00.475387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.475363    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" containerName="revoke-keys"
Oct 30 03:51:00.475537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.475390    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" containerName="revoke-keys"
Oct 30 03:51:00.475659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.475645    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" containerName="revoke-keys"
Oct 30 03:51:00.499826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.499791    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"]
Oct 30 03:51:00.500018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.499921    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.514033 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.514005    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:51:00.523659 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.523625    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:51:00.584298 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.584256    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.584588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.584396    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.584588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.584496    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q57q\" (UniqueName: \"kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.685987 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.685949    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.686240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.686056    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.686240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.686121    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q57q\" (UniqueName: \"kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.686941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.686908    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.689132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.689104    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.704333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.704301    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q57q\" (UniqueName: \"kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q\") pod \"maas-api-revoke-sub-2b29971b-bzb7t\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:00.838320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:00.838282    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:01.116147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.116064    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"]
Oct 30 03:51:01.153092 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.153061    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"]
Oct 30 03:51:01.153414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.153399    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.307970    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308079    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308154    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdvk\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308241 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308196    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308317    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308352    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308394    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308453    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.308783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.308758    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.410488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410392    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.410737 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410713    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.410816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410768    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.410874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410859    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdvk\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.410941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410904    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.410993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410974    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.411049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.410998    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.411049 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.411038    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.411159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.411061    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.411486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.411460    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.411874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.411851    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.411958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.411931    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.412609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.412584    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.412734 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.412628    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.413717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.413688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.415209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.415187    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.430651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.430626    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.431164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.431138    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdvk\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.453918 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.453870    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"]
Oct 30 03:51:01.455125 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:01.455093    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31b8385_d63e_47e3_8b43_72bd558b479b.slice/crio-50552e2f21714c1212a3722ba2b29f7b89f945d8e55135a26670114b51246957 WatchSource:0}: Error finding container 50552e2f21714c1212a3722ba2b29f7b89f945d8e55135a26670114b51246957: Status 404 returned error can't find the container with id 50552e2f21714c1212a3722ba2b29f7b89f945d8e55135a26670114b51246957
Oct 30 03:51:01.513973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.513930    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:01.820066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.820023    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t" event={"ID":"e31b8385-d63e-47e3-8b43-72bd558b479b","Type":"ContainerStarted","Data":"50552e2f21714c1212a3722ba2b29f7b89f945d8e55135a26670114b51246957"}
Oct 30 03:51:01.895777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.871052    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"]
Oct 30 03:51:01.895777 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:01.881902    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198b1ec6_ac60_49d8_bda5_581bbb36c219.slice/crio-287b37a8c006c7b915f6ac7d70d6d5558499f9bfdc676fd07f973af08e534a80 WatchSource:0}: Error finding container 287b37a8c006c7b915f6ac7d70d6d5558499f9bfdc676fd07f973af08e534a80: Status 404 returned error can't find the container with id 287b37a8c006c7b915f6ac7d70d6d5558499f9bfdc676fd07f973af08e534a80
Oct 30 03:51:01.895777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.886742    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:01.895777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.886834    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:01.895777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.886884    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:01.902783 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.899634    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"]
Oct 30 03:51:01.907415 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.906986    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.957565    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.957654    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.957806    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.957838    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.957876    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.958020    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.958049    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.958669    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.958792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.958760    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6hq\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:01.970066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:01.970036    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"]
Oct 30 03:51:02.059777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.059739    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.059947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.059818    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6hq\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.059947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.059895    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.059954    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060095    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060127    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060165    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060305 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060186    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060312    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060344    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060513    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060695    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060712    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.060943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.060888    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.062986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.062959    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.063892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.063871    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.078795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.078761    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6hq\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.079384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.079356    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.284722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.284671    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:02.616886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.616853    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"]
Oct 30 03:51:02.617305 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:02.617046    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13a6e55_0c25_48e7_a08c_765c3c57ef77.slice/crio-dc929491a6462896e836c492a2a7880507911e8e1f5e55dbcfdda1d2deffde1d WatchSource:0}: Error finding container dc929491a6462896e836c492a2a7880507911e8e1f5e55dbcfdda1d2deffde1d: Status 404 returned error can't find the container with id dc929491a6462896e836c492a2a7880507911e8e1f5e55dbcfdda1d2deffde1d
Oct 30 03:51:02.620389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.620357    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.620479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.620440    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.620523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.620484    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.836930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.836894    4877 generic.go:356] "Generic (PLEG): container finished" podID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerID="f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99" exitCode=0
Oct 30 03:51:02.837106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.837022    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" event={"ID":"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b","Type":"ContainerDied","Data":"f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"}
Oct 30 03:51:02.837664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.837631    4877 scope.go:122] "RemoveContainer" containerID="f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"
Oct 30 03:51:02.840525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.840348    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" event={"ID":"e13a6e55-0c25-48e7-a08c-765c3c57ef77","Type":"ContainerStarted","Data":"3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557"}
Oct 30 03:51:02.840525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.840407    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" event={"ID":"e13a6e55-0c25-48e7-a08c-765c3c57ef77","Type":"ContainerStarted","Data":"dc929491a6462896e836c492a2a7880507911e8e1f5e55dbcfdda1d2deffde1d"}
Oct 30 03:51:02.848131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.843718    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.848131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.843908    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.848131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.844061    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.850897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.850876    4877 generic.go:356] "Generic (PLEG): container finished" podID="e31b8385-d63e-47e3-8b43-72bd558b479b" containerID="d50c9bba711daaa87dd8796bb2d9ff40918fbcdf80adddd7e96a01083831fe1d" exitCode=0
Oct 30 03:51:02.851242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.851210    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t" event={"ID":"e31b8385-d63e-47e3-8b43-72bd558b479b","Type":"ContainerDied","Data":"d50c9bba711daaa87dd8796bb2d9ff40918fbcdf80adddd7e96a01083831fe1d"}
Oct 30 03:51:02.857479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.857447    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" event={"ID":"198b1ec6-ac60-49d8-bda5-581bbb36c219","Type":"ContainerStarted","Data":"e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775"}
Oct 30 03:51:02.857593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.857519    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" event={"ID":"198b1ec6-ac60-49d8-bda5-581bbb36c219","Type":"ContainerStarted","Data":"287b37a8c006c7b915f6ac7d70d6d5558499f9bfdc676fd07f973af08e534a80"}
Oct 30 03:51:02.866910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.866831    4877 generic.go:356] "Generic (PLEG): container finished" podID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerID="06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1" exitCode=0
Oct 30 03:51:02.866910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.866893    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" event={"ID":"e46386d6-80fd-40a9-aff1-1db047cfb8b9","Type":"ContainerDied","Data":"06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"}
Oct 30 03:51:02.867686 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.867663    4877 scope.go:122] "RemoveContainer" containerID="06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"
Oct 30 03:51:02.873799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.872058    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.873799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.873535    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.873799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.873596    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:02.998753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:02.973442    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podStartSLOduration=1.9734062159999999 podStartE2EDuration="1.973406216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:02.970885107 +0000 UTC m=+2891.399774548" watchObservedRunningTime="2026-10-30 03:51:02.973406216 +0000 UTC m=+2891.402295636"
Oct 30 03:51:03.143804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.143755    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podStartSLOduration=2.143740555 podStartE2EDuration="2.143740555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:03.139916678 +0000 UTC m=+2891.568806100" watchObservedRunningTime="2026-10-30 03:51:03.143740555 +0000 UTC m=+2891.572629972"
Oct 30 03:51:03.292534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.292497    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:03.310963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.310927    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:03.311258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.311226    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:03.518700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.518563    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:03.519276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.518955    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:03.519276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.519020    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:03.917857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.916391    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" event={"ID":"e46386d6-80fd-40a9-aff1-1db047cfb8b9","Type":"ContainerStarted","Data":"f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12"}
Oct 30 03:51:03.930280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:03.930241    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" event={"ID":"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b","Type":"ContainerStarted","Data":"b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586"}
Oct 30 03:51:04.238517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.237919    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:51:04.238517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.238069    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:04.238517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.238111    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:04.280449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.279867    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:51:04.311464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.294881    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:04.311464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.294950    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:04.477449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.473666    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:04.477449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.473883    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:04.477449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.471650    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:51:04.533536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.530195    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:04.533536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.530256    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:04.540459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:04.539510    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:51:05.277761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.277692    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:05.278413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.278337    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:05.286629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.286601    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:05.286852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.286826    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:05.788452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.745739    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:05.788452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.745800    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:05.788452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.770255    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:05.788452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.770310    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:05.832447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.831494    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 03:51:06.045536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:05.991149    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 03:51:06.257855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.257817    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:06.258140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.258112    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:06.310509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.291516    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:06.310509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.291574    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:06.336448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.325637    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:06.400088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.400045    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca\") pod \"e31b8385-d63e-47e3-8b43-72bd558b479b\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") "
Oct 30 03:51:06.400505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.400478    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca\") pod \"e31b8385-d63e-47e3-8b43-72bd558b479b\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") "
Oct 30 03:51:06.400846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.400824    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q\" (UniqueName: \"kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q\") pod \"e31b8385-d63e-47e3-8b43-72bd558b479b\" (UID: \"e31b8385-d63e-47e3-8b43-72bd558b479b\") "
Oct 30 03:51:06.452383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.451939    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca" pod "e31b8385-d63e-47e3-8b43-72bd558b479b" (UID: "e31b8385-d63e-47e3-8b43-72bd558b479b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:06.478533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.478473    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:06.622500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.478784    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.543517    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.547891    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.650066    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.567175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t" event={"ID":"e31b8385-d63e-47e3-8b43-72bd558b479b","Type":"ContainerDied","Data":"50552e2f21714c1212a3722ba2b29f7b89f945d8e55135a26670114b51246957"}
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.650146    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50552e2f21714c1212a3722ba2b29f7b89f945d8e55135a26670114b51246957"
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.567153    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"
Oct 30 03:51:06.674448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.604127    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q" pod "e31b8385-d63e-47e3-8b43-72bd558b479b" (UID: "e31b8385-d63e-47e3-8b43-72bd558b479b"). InnerVolumeSpecName "kube-api-access-2q57q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:06.801453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.719030    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca" pod "e31b8385-d63e-47e3-8b43-72bd558b479b" (UID: "e31b8385-d63e-47e3-8b43-72bd558b479b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:06.801453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.796598    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31b8385-d63e-47e3-8b43-72bd558b479b-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:06.801453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.796643    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q57q\" (UniqueName: \"kubernetes.io/projected/e31b8385-d63e-47e3-8b43-72bd558b479b-kube-api-access-2q57q\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:07.012506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:06.997780    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 03:51:07.038449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.013891    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:07.051479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.022830    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:07.062710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.062667    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:07.267651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.267563    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:07.267909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.267881    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:07.311441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.310706    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:07.311441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.310766    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:07.587956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.587914    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:07.588143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.587981    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:07.641453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.625380    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:07.660607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.660569    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:07.680255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.680150    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:07.976505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:07.976363    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:08.047851 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.047801    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:08.048863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.048838    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:51:08.061107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.061078    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:08.062571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.061723    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:51:08.366449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.320561    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.621274    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.549665    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.553936    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.621381    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.553975    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.621760    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.554039    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:08.628500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.621812    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:08.664728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.660472    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 03:51:08.664728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.660852    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:51:08.676473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.668496    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:08.988507 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.962592    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:08.990011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:08.989978    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:09.000755 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.000724    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:51:09.114611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.114560    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:51:09.180571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.180526    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:09.264910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.264703    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:09.264910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.264782    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:09.328582 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.328546    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:09.359933 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.329338    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:09.525188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.525153    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:09.531534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.531494    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:09.830632 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.830522    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:09.832290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.832254    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:09.852219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.852185    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 03:51:09.852458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.852414    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 03:51:09.945517 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.939880    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:51:09.966052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.965998    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 03:51:09.966735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:09.966710    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:51:10.014459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.003513    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:10.265523 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.248812    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:10.268860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.266292    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:10.272570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.267905    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 03:51:10.342706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.341663    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:10.342706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.342563    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:10.342706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.342603    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:10.450702 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.450611    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 03:51:10.465785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.465658    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:10.465785 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.465713    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:10.857444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.848577    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:10.857444 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.850147    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:10.900455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.886903    4877 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded"
Oct 30 03:51:10.930449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:10.928623    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 03:51:11.048476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.018908    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:11.464696 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.464507    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:11.466389 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.466368    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:11.475121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.475095    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:11.475308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.475285    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:11.475655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.475624    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 03:51:11.475932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.475902    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:11.501447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.476858    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 30 03:51:11.553240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.553195    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 30 03:51:11.553491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.502002    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:51:11.572888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.572708    4877 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 03:51:11.572888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.572763    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 03:51:11.850322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.849668    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:11.852626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.851437    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:11.912079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.909697    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 03:51:11.931528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.931484    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:51:11.942780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.942741    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 03:51:11.944124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.943782    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 03:51:11.961015 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.960755    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:51:11.975555 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:11.975529    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 03:51:12.236556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.236464    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 03:51:12.236556 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.236528    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 03:51:12.293934 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.293900    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:12.295121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.295099    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:12.295881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.295859    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:51:12.473767 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.473729    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:51:12.476348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:12.476312    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:51:13.236478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.235671    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"]
Oct 30 03:51:13.237266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.237008    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e31b8385-d63e-47e3-8b43-72bd558b479b" containerName="revoke-keys"
Oct 30 03:51:13.237266 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.237030    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31b8385-d63e-47e3-8b43-72bd558b479b" containerName="revoke-keys"
Oct 30 03:51:13.237380 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.237319    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31b8385-d63e-47e3-8b43-72bd558b479b" containerName="revoke-keys"
Oct 30 03:51:13.246502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.246311    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.280471 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.272043    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:51:13.284732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.284695    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:51:13.370446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.369476    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"]
Oct 30 03:51:13.489502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.476745    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.489502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.476845    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwvt\" (UniqueName: \"kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.489502 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.476881    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.570399 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.569940    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:51:13.575478 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.575041    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:51:13.579573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.579538    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.579864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.579840    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwvt\" (UniqueName: \"kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.580011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.579997    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.581102 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.581081    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.589220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.589169    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.607188 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.607162    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwvt\" (UniqueName: \"kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt\") pod \"maas-api-revoke-sub-edf68f22-vwr4z\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:13.637953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:13.637926    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:14.202231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:14.202203    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"]
Oct 30 03:51:14.602040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:14.602005    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z" event={"ID":"5771a794-0c7b-4d0b-b327-368d8af9980e","Type":"ContainerStarted","Data":"deaf362b3ea4a09c9eb399844c4ceacd78658434406544e2b58d3d9cfc13959d"}
Oct 30 03:51:14.650855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:14.650801    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z" podStartSLOduration=1.6507867969999999 podStartE2EDuration="1.650786797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:14.645512638 +0000 UTC m=+2903.074402056" watchObservedRunningTime="2026-10-30 03:51:14.650786797 +0000 UTC m=+2903.079676214"
Oct 30 03:51:15.615565 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:15.615526    4877 generic.go:356] "Generic (PLEG): container finished" podID="5771a794-0c7b-4d0b-b327-368d8af9980e" containerID="4491a1da8a2405ea00389bc2b1b3de825addbd023efbee4a00a723976db79eb5" exitCode=0
Oct 30 03:51:15.616078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:15.615605    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z" event={"ID":"5771a794-0c7b-4d0b-b327-368d8af9980e","Type":"ContainerDied","Data":"4491a1da8a2405ea00389bc2b1b3de825addbd023efbee4a00a723976db79eb5"}
Oct 30 03:51:16.232702 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:51:16.232667    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:51:16.880476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.880401    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:16.978075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.978044    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt\" (UniqueName: \"kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt\") pod \"5771a794-0c7b-4d0b-b327-368d8af9980e\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") "
Oct 30 03:51:16.978239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.978110    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca\") pod \"5771a794-0c7b-4d0b-b327-368d8af9980e\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") "
Oct 30 03:51:16.978286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.978265    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca\") pod \"5771a794-0c7b-4d0b-b327-368d8af9980e\" (UID: \"5771a794-0c7b-4d0b-b327-368d8af9980e\") "
Oct 30 03:51:16.978684 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.978656    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca" pod "5771a794-0c7b-4d0b-b327-368d8af9980e" (UID: "5771a794-0c7b-4d0b-b327-368d8af9980e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:16.978998 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.978979    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:16.980374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.980344    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt" pod "5771a794-0c7b-4d0b-b327-368d8af9980e" (UID: "5771a794-0c7b-4d0b-b327-368d8af9980e"). InnerVolumeSpecName "kube-api-access-9pwvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:16.980715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:16.980695    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca" pod "5771a794-0c7b-4d0b-b327-368d8af9980e" (UID: "5771a794-0c7b-4d0b-b327-368d8af9980e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:17.080818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:17.080765    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwvt\" (UniqueName: \"kubernetes.io/projected/5771a794-0c7b-4d0b-b327-368d8af9980e-kube-api-access-9pwvt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:17.080818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:17.080817    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5771a794-0c7b-4d0b-b327-368d8af9980e-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:17.635398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:17.635369    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"
Oct 30 03:51:17.635690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:17.635394    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z" event={"ID":"5771a794-0c7b-4d0b-b327-368d8af9980e","Type":"ContainerDied","Data":"deaf362b3ea4a09c9eb399844c4ceacd78658434406544e2b58d3d9cfc13959d"}
Oct 30 03:51:17.635690 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:17.635466    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deaf362b3ea4a09c9eb399844c4ceacd78658434406544e2b58d3d9cfc13959d"
Oct 30 03:51:19.028644 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:19.028604    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"]
Oct 30 03:51:19.037341 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:19.037317    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec9a34fa-zcpqq"]
Oct 30 03:51:19.702864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:19.702831    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de46bdd6-4804-4996-8929-e8023626beaf" path="/var/lib/kubelet/pods/de46bdd6-4804-4996-8929-e8023626beaf/volumes"
Oct 30 03:51:21.035481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:21.035443    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"]
Oct 30 03:51:21.043123 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:21.043097    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5204ed74-smlkt"]
Oct 30 03:51:21.710853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:21.710650    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0" path="/var/lib/kubelet/pods/07c32834-13e1-46ea-bb9f-ff6bbd3a5bb0/volumes"
Oct 30 03:51:23.931960 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.931924    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"]
Oct 30 03:51:23.932996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.932970    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5771a794-0c7b-4d0b-b327-368d8af9980e" containerName="revoke-keys"
Oct 30 03:51:23.932996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.932997    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5771a794-0c7b-4d0b-b327-368d8af9980e" containerName="revoke-keys"
Oct 30 03:51:23.933260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.933244    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5771a794-0c7b-4d0b-b327-368d8af9980e" containerName="revoke-keys"
Oct 30 03:51:23.937499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.937474    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:23.955395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.955367    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"]
Oct 30 03:51:23.958416 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.958393    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:51:23.958538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:23.958509    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:51:24.032758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.032717    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gvt\" (UniqueName: \"kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.032958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.032927    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.033087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.033067    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.135058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.135024    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.135239 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.135156    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.135347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.135328    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gvt\" (UniqueName: \"kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.135900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.135869    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.137708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.137683    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.153202 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.153170    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gvt\" (UniqueName: \"kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt\") pod \"maas-api-revoke-sub-0f1bafeb-6bd6k\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.254849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.254761    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:24.574925 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:24.574895    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d5571e_a3b5_46fe_ad7a_2dc47cf50bf2.slice/crio-076fe5dc1f1c2ae465dcd90b1ff74e930bd45577f8ffb5e1f03cf106748bd01b WatchSource:0}: Error finding container 076fe5dc1f1c2ae465dcd90b1ff74e930bd45577f8ffb5e1f03cf106748bd01b: Status 404 returned error can't find the container with id 076fe5dc1f1c2ae465dcd90b1ff74e930bd45577f8ffb5e1f03cf106748bd01b
Oct 30 03:51:24.575387 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.575365    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"]
Oct 30 03:51:24.707809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:24.707543    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k" event={"ID":"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2","Type":"ContainerStarted","Data":"076fe5dc1f1c2ae465dcd90b1ff74e930bd45577f8ffb5e1f03cf106748bd01b"}
Oct 30 03:51:25.719530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:25.719487    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k" event={"ID":"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2","Type":"ContainerStarted","Data":"65994b9f200eee8416cc816b484bd2d0bbb71610166f4ffda843f75d10f8e949"}
Oct 30 03:51:25.756827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:25.756786    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k" podStartSLOduration=2.756772868 podStartE2EDuration="2.756772868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:25.7528725 +0000 UTC m=+2914.181761919" watchObservedRunningTime="2026-10-30 03:51:25.756772868 +0000 UTC m=+2914.185662286"
Oct 30 03:51:26.731718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:26.731679    4877 generic.go:356] "Generic (PLEG): container finished" podID="16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" containerID="65994b9f200eee8416cc816b484bd2d0bbb71610166f4ffda843f75d10f8e949" exitCode=0
Oct 30 03:51:26.732125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:26.731753    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k" event={"ID":"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2","Type":"ContainerDied","Data":"65994b9f200eee8416cc816b484bd2d0bbb71610166f4ffda843f75d10f8e949"}
Oct 30 03:51:27.642344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:27.642165    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"]
Oct 30 03:51:27.946347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:27.946317    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:28.137125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.137087    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca\") pod \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") "
Oct 30 03:51:28.137293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.137261    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt\" (UniqueName: \"kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt\") pod \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") "
Oct 30 03:51:28.137692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.137663    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca\") pod \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\" (UID: \"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2\") "
Oct 30 03:51:28.138784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.138131    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca" pod "16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" (UID: "16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:28.138784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.138562    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:28.139981 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.139953    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca" pod "16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" (UID: "16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:28.140361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.140329    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt" pod "16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" (UID: "16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2"). InnerVolumeSpecName "kube-api-access-z4gvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:28.282447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.279927    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:28.282447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.279974    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4gvt\" (UniqueName: \"kubernetes.io/projected/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2-kube-api-access-z4gvt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:28.751158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.751082    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"
Oct 30 03:51:28.751158 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.751139    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k" event={"ID":"16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2","Type":"ContainerDied","Data":"076fe5dc1f1c2ae465dcd90b1ff74e930bd45577f8ffb5e1f03cf106748bd01b"}
Oct 30 03:51:28.751362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:28.751196    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="076fe5dc1f1c2ae465dcd90b1ff74e930bd45577f8ffb5e1f03cf106748bd01b"
Oct 30 03:51:32.643170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:32.643120    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w" podUID="739f1aec-7e90-4787-80e8-18d13f43849e" containerName="payload-processing" containerID="cri-o://baddcc94b39bdcd3d32ef1f511a1000c7d61294b1c767a5fc6697bb83bd6e0a3" gracePeriod=25
Oct 30 03:51:32.792600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:32.792564    4877 generic.go:356] "Generic (PLEG): container finished" podID="739f1aec-7e90-4787-80e8-18d13f43849e" containerID="baddcc94b39bdcd3d32ef1f511a1000c7d61294b1c767a5fc6697bb83bd6e0a3" exitCode=0
Oct 30 03:51:32.792749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:32.792648    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w" event={"ID":"739f1aec-7e90-4787-80e8-18d13f43849e","Type":"ContainerDied","Data":"baddcc94b39bdcd3d32ef1f511a1000c7d61294b1c767a5fc6697bb83bd6e0a3"}
Oct 30 03:51:32.957571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:32.957541    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:51:33.065525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.065490    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg\") pod \"739f1aec-7e90-4787-80e8-18d13f43849e\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") "
Oct 30 03:51:33.065705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.065601    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials\") pod \"739f1aec-7e90-4787-80e8-18d13f43849e\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") "
Oct 30 03:51:33.065750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.065742    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume\") pod \"739f1aec-7e90-4787-80e8-18d13f43849e\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") "
Oct 30 03:51:33.065853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.065831    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume\") pod \"739f1aec-7e90-4787-80e8-18d13f43849e\" (UID: \"739f1aec-7e90-4787-80e8-18d13f43849e\") "
Oct 30 03:51:33.066195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.066167    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume" pod "739f1aec-7e90-4787-80e8-18d13f43849e" (UID: "739f1aec-7e90-4787-80e8-18d13f43849e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:33.066329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.066224    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume" pod "739f1aec-7e90-4787-80e8-18d13f43849e" (UID: "739f1aec-7e90-4787-80e8-18d13f43849e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:33.066570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.066553    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:33.066677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.066572    4877 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739f1aec-7e90-4787-80e8-18d13f43849e-routing-overlay-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:33.067753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.067729    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials" pod "739f1aec-7e90-4787-80e8-18d13f43849e" (UID: "739f1aec-7e90-4787-80e8-18d13f43849e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:33.068126 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.068100    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg" pod "739f1aec-7e90-4787-80e8-18d13f43849e" (UID: "739f1aec-7e90-4787-80e8-18d13f43849e"). InnerVolumeSpecName "kube-api-access-r45wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:33.168349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.168315    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r45wg\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-kube-api-access-r45wg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:33.168349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.168345    4877 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739f1aec-7e90-4787-80e8-18d13f43849e-provider-credentials\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:51:33.803153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.803121    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"
Oct 30 03:51:33.803653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.803174    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w" event={"ID":"739f1aec-7e90-4787-80e8-18d13f43849e","Type":"ContainerDied","Data":"02f49d4487a48c64fa2bde2fb568d469819bb54d4a630c0419cf104af12301be"}
Oct 30 03:51:33.803653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.803251    4877 scope.go:122] "RemoveContainer" containerID="baddcc94b39bdcd3d32ef1f511a1000c7d61294b1c767a5fc6697bb83bd6e0a3"
Oct 30 03:51:33.851021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.850992    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"]
Oct 30 03:51:33.863357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:33.863331    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-68b666l6w"]
Oct 30 03:51:35.704175 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:35.704137    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739f1aec-7e90-4787-80e8-18d13f43849e" path="/var/lib/kubelet/pods/739f1aec-7e90-4787-80e8-18d13f43849e/volumes"
Oct 30 03:51:43.401234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.401189    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"]
Oct 30 03:51:43.403054 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.403027    4877 cpu_manager.go:414] "containerMap: removing container" podUID="739f1aec-7e90-4787-80e8-18d13f43849e" containerName="payload-processing"
Oct 30 03:51:43.403201 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.403187    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739f1aec-7e90-4787-80e8-18d13f43849e" containerName="payload-processing"
Oct 30 03:51:43.403346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.403336    4877 cpu_manager.go:414] "containerMap: removing container" podUID="16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" containerName="revoke-keys"
Oct 30 03:51:43.403543 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.403501    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" containerName="revoke-keys"
Oct 30 03:51:43.403961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.403941    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" containerName="revoke-keys"
Oct 30 03:51:43.409706 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.409688    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="739f1aec-7e90-4787-80e8-18d13f43849e" containerName="payload-processing"
Oct 30 03:51:43.507539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.506954    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.516819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.516776    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.516840    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.516889    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.516932    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlc7g\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517018 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.516968    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.517032    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.517286    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.517344    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.517405 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.517380    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.523809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.523723    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a3d585-openshift-default-dockercfg-5nbk9\""
Oct 30 03:51:43.539023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.538977    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"]
Oct 30 03:51:43.629938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.629897    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlc7g\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653293    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653442    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653769    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653814    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653927    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.653981    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.654500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.654047    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.655093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.654578    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.655093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.654849    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.658245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.658212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.659564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.659534    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.689504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.681747    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.712443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.710889    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.712443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.711137    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.712443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.712228    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.724445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.724182    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlc7g\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g\") pod \"e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") " pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:43.847446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:43.844820    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:44.275037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:44.271056    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"]
Oct 30 03:51:44.279278 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:44.279242    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd276f194_aaa3_4df4_8104_e8844bb5aa82.slice/crio-4246f80468656b2ff8c1cb66d87501e796a320b30aa3c06696af38cc2c434246 WatchSource:0}: Error finding container 4246f80468656b2ff8c1cb66d87501e796a320b30aa3c06696af38cc2c434246: Status 404 returned error can't find the container with id 4246f80468656b2ff8c1cb66d87501e796a320b30aa3c06696af38cc2c434246
Oct 30 03:51:44.283106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:44.283065    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:44.283245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:44.283169    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:44.283245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:44.283208    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:45.155448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:45.153022    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" event={"ID":"d276f194-aaa3-4df4-8104-e8844bb5aa82","Type":"ContainerStarted","Data":"fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"}
Oct 30 03:51:45.155448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:45.154837    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" event={"ID":"d276f194-aaa3-4df4-8104-e8844bb5aa82","Type":"ContainerStarted","Data":"4246f80468656b2ff8c1cb66d87501e796a320b30aa3c06696af38cc2c434246"}
Oct 30 03:51:45.247442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:45.240193    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podStartSLOduration=2.240169024 podStartE2EDuration="2.240169024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:45.228165286 +0000 UTC m=+2933.657054714" watchObservedRunningTime="2026-10-30 03:51:45.240169024 +0000 UTC m=+2933.669058446"
Oct 30 03:51:45.855491 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:45.849359    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:45.877446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:45.859793    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:46.197508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:46.197349    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:46.232596 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:46.232551    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:51:46.264257 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:51:46.264214    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:51:49.645536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:49.645493    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"]
Oct 30 03:51:49.651670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:49.651651    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:49.675196 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:49.675165    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a3d585-dockercfg-qnn6t\""
Oct 30 03:51:49.675336 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:49.675240    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a3d585\""
Oct 30 03:51:50.170456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.169094    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896jf\" (UniqueName: \"kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf\") pod \"payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:50.170456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.170320    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"]
Oct 30 03:51:50.170746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.170579    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:50.240571 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.240534    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"]
Oct 30 03:51:50.246961 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.246938    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"]
Oct 30 03:51:50.247091 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.247025    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.272234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.272209    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896jf\" (UniqueName: \"kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf\") pod \"payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:50.272372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.272354    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:50.273170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.273142    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:50.323191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.323155    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896jf\" (UniqueName: \"kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf\") pod \"payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:50.373668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.373635    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.373846 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.373681    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmv2\" (UniqueName: \"kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2\") pod \"payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.475619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.475532    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.475619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.475578    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmv2\" (UniqueName: \"kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2\") pod \"payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.476232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.476212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.516801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.516405    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmv2\" (UniqueName: \"kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2\") pod \"payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.568761 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.562724    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:50.569569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:50.569538    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:51.013583 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:51.013538    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24f7bc8_24b0_463b_9667_450610471d42.slice/crio-8ea10bcdc1d25d1ae6e0f46e7f1485c06e9fbaa6b82c8523cc05c2c1e157a04e WatchSource:0}: Error finding container 8ea10bcdc1d25d1ae6e0f46e7f1485c06e9fbaa6b82c8523cc05c2c1e157a04e: Status 404 returned error can't find the container with id 8ea10bcdc1d25d1ae6e0f46e7f1485c06e9fbaa6b82c8523cc05c2c1e157a04e
Oct 30 03:51:51.017476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.017445    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"]
Oct 30 03:51:51.022648 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:51:51.022619    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cf83a4_3795_43c8_86b7_3c5a4c9a2090.slice/crio-bdb0a41b7d9ee06673f20f73d6b0ff2cd7e2f46ce8fdfc4e55ebef582983632f WatchSource:0}: Error finding container bdb0a41b7d9ee06673f20f73d6b0ff2cd7e2f46ce8fdfc4e55ebef582983632f: Status 404 returned error can't find the container with id bdb0a41b7d9ee06673f20f73d6b0ff2cd7e2f46ce8fdfc4e55ebef582983632f
Oct 30 03:51:51.027908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.027883    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"]
Oct 30 03:51:51.040046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.040021    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"]
Oct 30 03:51:51.046079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.046046    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39da9501-7dm25"]
Oct 30 03:51:51.256439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.256274    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" event={"ID":"a24f7bc8-24b0-463b-9667-450610471d42","Type":"ContainerStarted","Data":"ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065"}
Oct 30 03:51:51.256439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.256325    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" event={"ID":"a24f7bc8-24b0-463b-9667-450610471d42","Type":"ContainerStarted","Data":"8ea10bcdc1d25d1ae6e0f46e7f1485c06e9fbaa6b82c8523cc05c2c1e157a04e"}
Oct 30 03:51:51.257179 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.257161    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:51:51.259020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.258999    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" event={"ID":"28cf83a4-3795-43c8-86b7-3c5a4c9a2090","Type":"ContainerStarted","Data":"ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279"}
Oct 30 03:51:51.259088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.259040    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" event={"ID":"28cf83a4-3795-43c8-86b7-3c5a4c9a2090","Type":"ContainerStarted","Data":"bdb0a41b7d9ee06673f20f73d6b0ff2cd7e2f46ce8fdfc4e55ebef582983632f"}
Oct 30 03:51:51.259625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.259583    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:51:51.301458 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.298003    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" podStartSLOduration=2.29798722 podStartE2EDuration="2.29798722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:51.296691249 +0000 UTC m=+2939.725580668" watchObservedRunningTime="2026-10-30 03:51:51.29798722 +0000 UTC m=+2939.726876632"
Oct 30 03:51:51.343398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.343300    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" podStartSLOduration=2.343283576 podStartE2EDuration="2.343283576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:51.334947726 +0000 UTC m=+2939.763837138" watchObservedRunningTime="2026-10-30 03:51:51.343283576 +0000 UTC m=+2939.772172995"
Oct 30 03:51:51.707108 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:51.707066    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c255e4-a5f3-40d6-9742-13ae33d599ee" path="/var/lib/kubelet/pods/08c255e4-a5f3-40d6-9742-13ae33d599ee/volumes"
Oct 30 03:51:55.666020 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:55.665978    4877 scope.go:122] "RemoveContainer" containerID="3b19b969fe6c493fd7f696138dfa6925feb0a0450ba3f3f77dd49ec859611a50"
Oct 30 03:51:55.703639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:55.703610    4877 scope.go:122] "RemoveContainer" containerID="f362e7bb7c550ddc372872f48739e85eda9552e5d3a718abb99003c59544732e"
Oct 30 03:51:56.317697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:56.317660    4877 generic.go:356] "Generic (PLEG): container finished" podID="e3660296-8703-4234-84cd-c8b96142c79f" containerID="1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788" exitCode=0
Oct 30 03:51:56.317957 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:56.317743    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" event={"ID":"e3660296-8703-4234-84cd-c8b96142c79f","Type":"ContainerDied","Data":"1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"}
Oct 30 03:51:56.318236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:56.318219    4877 scope.go:122] "RemoveContainer" containerID="1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"
Oct 30 03:51:56.320319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:56.320286    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:56.320473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:56.320359    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:56.320473 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:56.320390    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:51:57.329390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:57.329350    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" event={"ID":"e3660296-8703-4234-84cd-c8b96142c79f","Type":"ContainerStarted","Data":"13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5"}
Oct 30 03:51:57.989446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:57.986781    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:51:58.002453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:57.995531    4877 patch_prober.go:29] interesting pod/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 03:51:58.002453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:57.995597    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 03:51:58.982457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:58.980759    4877 patch_prober.go:29] interesting pod/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 03:51:58.982457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:51:58.980832    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 03:52:00.003448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:00.002562    4877 patch_prober.go:29] interesting pod/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 03:52:00.003448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:00.002638    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 03:52:00.135559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:00.135516    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:52:00.980825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:00.980782    4877 patch_prober.go:29] interesting pod/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 03:52:00.981139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:00.981106    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 03:52:01.980865 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:01.980823    4877 patch_prober.go:29] interesting pod/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 30 03:52:01.981499 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:01.981465    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 30 03:52:02.996492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:02.996455    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:52:03.000263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:03.000237    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:52:03.285978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:03.285947    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 03:52:03.286184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:03.286017    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 03:52:10.048465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.048413    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"]
Oct 30 03:52:10.056043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.056014    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.069877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.069847    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9e0d09-openshift-default-dockercfg-kfh5d\""
Oct 30 03:52:10.091623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.091584    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"]
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.256967    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257170    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257249    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257329    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257369    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257461    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257524    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257568    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.257715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.257714    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhng\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.362309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.362215    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.362640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.362611    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.362854 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.362835    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.363006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.362989    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.363164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.363149    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.363311 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.363295    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.363460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.363443    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.363710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.363687    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhng\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.363858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.363838    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.365007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.364984    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.365407 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.365385    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.374250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.374220    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.406661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.375810    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.406929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.376045    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.428390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.417112    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhng\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.433930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.433800    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.435585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.435557    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.438383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.438337    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:10.686494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:10.685751    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:11.105799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.105768    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"]
Oct 30 03:52:11.106808 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:52:11.106782    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda633a9a2_e8ac_415b_95c3_4d478846e0ce.slice/crio-2c6c451808aa243a5083fd4a300067495fad6b1dbee7f1cdd80e7bf89300214e WatchSource:0}: Error finding container 2c6c451808aa243a5083fd4a300067495fad6b1dbee7f1cdd80e7bf89300214e: Status 404 returned error can't find the container with id 2c6c451808aa243a5083fd4a300067495fad6b1dbee7f1cdd80e7bf89300214e
Oct 30 03:52:11.110181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.110142    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:52:11.110301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.110215    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:52:11.110301 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.110244    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:52:11.603149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.603103    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" event={"ID":"a633a9a2-e8ac-415b-95c3-4d478846e0ce","Type":"ContainerStarted","Data":"d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"}
Oct 30 03:52:11.603364 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.603174    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" event={"ID":"a633a9a2-e8ac-415b-95c3-4d478846e0ce","Type":"ContainerStarted","Data":"2c6c451808aa243a5083fd4a300067495fad6b1dbee7f1cdd80e7bf89300214e"}
Oct 30 03:52:11.653313 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.653261    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podStartSLOduration=1.653247424 podStartE2EDuration="1.653247424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:11.649001995 +0000 UTC m=+2960.077891413" watchObservedRunningTime="2026-10-30 03:52:11.653247424 +0000 UTC m=+2960.082136841"
Oct 30 03:52:11.686876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:11.686837    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:12.693825 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:12.693794    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:13.626162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:13.626129    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:13.627186 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:13.627157    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:52:15.979351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:15.979314    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"]
Oct 30 03:52:15.984628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:15.984605    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.004140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.004110    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"]
Oct 30 03:52:16.005402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.005377    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9e0d09\""
Oct 30 03:52:16.005595 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.005491    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9e0d09-dockercfg-cgdr7\""
Oct 30 03:52:16.067651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.067610    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.067840 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.067664    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzx6j\" (UniqueName: \"kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j\") pod \"payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.245638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.232999    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.245638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.233079    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzx6j\" (UniqueName: \"kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j\") pod \"payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.246850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.246764    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.432513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.403177    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzx6j\" (UniqueName: \"kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j\") pod \"payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.553620 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:52:16.553581    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:52:16.602308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.602261    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:16.911841 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:52:16.911809    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bc4dcf_f2b4_461a_9cda_5430dc76e21a.slice/crio-563bf196e59e620183a61c3d131eb960a3034d10eac02a5293028c5a37000f0f WatchSource:0}: Error finding container 563bf196e59e620183a61c3d131eb960a3034d10eac02a5293028c5a37000f0f: Status 404 returned error can't find the container with id 563bf196e59e620183a61c3d131eb960a3034d10eac02a5293028c5a37000f0f
Oct 30 03:52:16.915086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:16.915057    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"]
Oct 30 03:52:17.701004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:17.700970    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp" event={"ID":"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a","Type":"ContainerStarted","Data":"1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e"}
Oct 30 03:52:17.701004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:17.701011    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:17.701479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:17.701026    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp" event={"ID":"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a","Type":"ContainerStarted","Data":"563bf196e59e620183a61c3d131eb960a3034d10eac02a5293028c5a37000f0f"}
Oct 30 03:52:17.732773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:17.732721    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp" podStartSLOduration=2.7327078289999998 podStartE2EDuration="2.732707829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:17.728505403 +0000 UTC m=+2966.157394838" watchObservedRunningTime="2026-10-30 03:52:17.732707829 +0000 UTC m=+2966.161597247"
Oct 30 03:52:24.030747 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:24.030711    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"]
Oct 30 03:52:24.039884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:24.039857    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f2a4f0-sb5j6"]
Oct 30 03:52:25.028606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:25.028574    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"]
Oct 30 03:52:25.033807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:25.033778    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97066d73-vbwp8"]
Oct 30 03:52:25.700656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:25.700617    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f6d363-edf0-477b-b114-6d80b7d192b5" path="/var/lib/kubelet/pods/46f6d363-edf0-477b-b114-6d80b7d192b5/volumes"
Oct 30 03:52:25.701205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:25.701184    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f315c41-cf78-483e-9d57-2237b2d9ee12" path="/var/lib/kubelet/pods/5f315c41-cf78-483e-9d57-2237b2d9ee12/volumes"
Oct 30 03:52:26.534892 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.534789    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"]
Oct 30 03:52:26.544504 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.544477    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.577597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.566806    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:52:26.577597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.566875    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:52:26.577597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.571611    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"]
Oct 30 03:52:26.679248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.678262    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.679814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.679788    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wt4v\" (UniqueName: \"kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.680246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.680225    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.782039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.781998    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.782238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.782174    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.782329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.782311    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wt4v\" (UniqueName: \"kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.782703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.782679    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.784735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.784715    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.799652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.799600    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wt4v\" (UniqueName: \"kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v\") pod \"maas-api-revoke-sub-62ece13c-fzr2h\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:26.978043 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:26.978005    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:27.290092 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:52:27.290057    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1cb12d_fc8e_4ea3_a319_2678d2a1f249.slice/crio-b63266bc579b11136fc4034bdda0bd9f4574bf552a6ac1911574e8849cb7f9ad WatchSource:0}: Error finding container b63266bc579b11136fc4034bdda0bd9f4574bf552a6ac1911574e8849cb7f9ad: Status 404 returned error can't find the container with id b63266bc579b11136fc4034bdda0bd9f4574bf552a6ac1911574e8849cb7f9ad
Oct 30 03:52:27.291697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:27.291659    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"]
Oct 30 03:52:27.924487 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:27.924454    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h" event={"ID":"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249","Type":"ContainerStarted","Data":"6ac146966556d3e49208ffd9e580ea46c4374f537320532b76d1dcf50bf6dabd"}
Oct 30 03:52:27.924904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:27.924505    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h" event={"ID":"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249","Type":"ContainerStarted","Data":"b63266bc579b11136fc4034bdda0bd9f4574bf552a6ac1911574e8849cb7f9ad"}
Oct 30 03:52:27.957882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:27.957800    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h" podStartSLOduration=1.957786895 podStartE2EDuration="1.957786895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:27.956697177 +0000 UTC m=+2976.385586623" watchObservedRunningTime="2026-10-30 03:52:27.957786895 +0000 UTC m=+2976.386676367"
Oct 30 03:52:28.708232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:28.708155    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:52:28.936719 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:28.936683    4877 generic.go:356] "Generic (PLEG): container finished" podID="4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" containerID="6ac146966556d3e49208ffd9e580ea46c4374f537320532b76d1dcf50bf6dabd" exitCode=0
Oct 30 03:52:28.937237 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:28.936748    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h" event={"ID":"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249","Type":"ContainerDied","Data":"6ac146966556d3e49208ffd9e580ea46c4374f537320532b76d1dcf50bf6dabd"}
Oct 30 03:52:30.143220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.143187    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:30.243700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.243600    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v\" (UniqueName: \"kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v\") pod \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") "
Oct 30 03:52:30.243884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.243717    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca\") pod \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") "
Oct 30 03:52:30.244029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.244006    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca\") pod \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\" (UID: \"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249\") "
Oct 30 03:52:30.244362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.244336    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca" pod "4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" (UID: "4c1cb12d-fc8e-4ea3-a319-2678d2a1f249"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:52:30.244729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.244711    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:30.246350 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.246322    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v" pod "4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" (UID: "4c1cb12d-fc8e-4ea3-a319-2678d2a1f249"). InnerVolumeSpecName "kube-api-access-8wt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:52:30.246464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.246332    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca" pod "4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" (UID: "4c1cb12d-fc8e-4ea3-a319-2678d2a1f249"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:52:30.346164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.346113    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:30.346164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.346143    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wt4v\" (UniqueName: \"kubernetes.io/projected/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249-kube-api-access-8wt4v\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:30.968613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.968581    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"
Oct 30 03:52:30.968613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.968606    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h" event={"ID":"4c1cb12d-fc8e-4ea3-a319-2678d2a1f249","Type":"ContainerDied","Data":"b63266bc579b11136fc4034bdda0bd9f4574bf552a6ac1911574e8849cb7f9ad"}
Oct 30 03:52:30.968861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:30.968665    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b63266bc579b11136fc4034bdda0bd9f4574bf552a6ac1911574e8849cb7f9ad"
Oct 30 03:52:36.030143 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:36.030101    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"]
Oct 30 03:52:36.037498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:36.037466    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b457f708-chdkw"]
Oct 30 03:52:37.218248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.218214    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"]
Oct 30 03:52:37.219122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.219103    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" containerName="revoke-keys"
Oct 30 03:52:37.219173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.219125    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" containerName="revoke-keys"
Oct 30 03:52:37.219300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.219290    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" containerName="revoke-keys"
Oct 30 03:52:37.223634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.223613    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.252448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.242854    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"]
Oct 30 03:52:37.264603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.264563    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:52:37.265026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.265002    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:52:37.384602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.384560    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjpb\" (UniqueName: \"kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.384779 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.384731    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.384876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.384764    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.486436 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.486331    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjpb\" (UniqueName: \"kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.486680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.486480    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.486680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.486508    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.487280 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.487250    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.489048 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.489024    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.503357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.503326    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjpb\" (UniqueName: \"kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb\") pod \"maas-api-revoke-sub-24a4f986-plmg8\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.560457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.560419    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:37.708231 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.708191    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f50ae9-00d4-450a-a8c7-97e6463378d7" path="/var/lib/kubelet/pods/16f50ae9-00d4-450a-a8c7-97e6463378d7/volumes"
Oct 30 03:52:37.872301 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:52:37.872273    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f36106_0c2e_42ca_8654_15b263f94696.slice/crio-b8cbab3a2c1aba54dc6793013468b7d95b51e25b37ff66c1dd6c4fb54d4a5a45 WatchSource:0}: Error finding container b8cbab3a2c1aba54dc6793013468b7d95b51e25b37ff66c1dd6c4fb54d4a5a45: Status 404 returned error can't find the container with id b8cbab3a2c1aba54dc6793013468b7d95b51e25b37ff66c1dd6c4fb54d4a5a45
Oct 30 03:52:37.873695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:37.873665    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"]
Oct 30 03:52:38.051978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:38.051943    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8" event={"ID":"36f36106-0c2e-42ca-8654-15b263f94696","Type":"ContainerStarted","Data":"b8cbab3a2c1aba54dc6793013468b7d95b51e25b37ff66c1dd6c4fb54d4a5a45"}
Oct 30 03:52:39.064551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:39.064465    4877 generic.go:356] "Generic (PLEG): container finished" podID="36f36106-0c2e-42ca-8654-15b263f94696" containerID="b4ed20ec24f3b527628cc6c8b10852c87d5ef0ff1255fb5d8f25cb6912abe3f7" exitCode=0
Oct 30 03:52:39.064919 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:39.064547    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8" event={"ID":"36f36106-0c2e-42ca-8654-15b263f94696","Type":"ContainerDied","Data":"b4ed20ec24f3b527628cc6c8b10852c87d5ef0ff1255fb5d8f25cb6912abe3f7"}
Oct 30 03:52:40.360860 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.360834    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:40.447135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.447098    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca\") pod \"36f36106-0c2e-42ca-8654-15b263f94696\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") "
Oct 30 03:52:40.447567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.447538    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca\") pod \"36f36106-0c2e-42ca-8654-15b263f94696\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") "
Oct 30 03:52:40.447725 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.447630    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb\" (UniqueName: \"kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb\") pod \"36f36106-0c2e-42ca-8654-15b263f94696\" (UID: \"36f36106-0c2e-42ca-8654-15b263f94696\") "
Oct 30 03:52:40.447838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.447546    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca" pod "36f36106-0c2e-42ca-8654-15b263f94696" (UID: "36f36106-0c2e-42ca-8654-15b263f94696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:52:40.448394 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.448360    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:40.450528 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.450501    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca" pod "36f36106-0c2e-42ca-8654-15b263f94696" (UID: "36f36106-0c2e-42ca-8654-15b263f94696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:52:40.452929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.452891    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb" pod "36f36106-0c2e-42ca-8654-15b263f94696" (UID: "36f36106-0c2e-42ca-8654-15b263f94696"). InnerVolumeSpecName "kube-api-access-xwjpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:52:40.549640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.549613    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36f36106-0c2e-42ca-8654-15b263f94696-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:40.549640 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:40.549638    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjpb\" (UniqueName: \"kubernetes.io/projected/36f36106-0c2e-42ca-8654-15b263f94696-kube-api-access-xwjpb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:41.084786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:41.084754    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"
Oct 30 03:52:41.084985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:41.084791    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8" event={"ID":"36f36106-0c2e-42ca-8654-15b263f94696","Type":"ContainerDied","Data":"b8cbab3a2c1aba54dc6793013468b7d95b51e25b37ff66c1dd6c4fb54d4a5a45"}
Oct 30 03:52:41.084985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:41.084839    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8cbab3a2c1aba54dc6793013468b7d95b51e25b37ff66c1dd6c4fb54d4a5a45"
Oct 30 03:52:46.231573 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:52:46.231534    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:52:50.500490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.499986    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"]
Oct 30 03:52:50.501971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.501913    4877 cpu_manager.go:414] "containerMap: removing container" podUID="36f36106-0c2e-42ca-8654-15b263f94696" containerName="revoke-keys"
Oct 30 03:52:50.501971 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.501940    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f36106-0c2e-42ca-8654-15b263f94696" containerName="revoke-keys"
Oct 30 03:52:50.502248 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.502226    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f36106-0c2e-42ca-8654-15b263f94696" containerName="revoke-keys"
Oct 30 03:52:50.508888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.508865    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.511156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.511129    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"]
Oct 30 03:52:50.522084 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.522060    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:52:50.532031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.532007    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:52:50.602078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.602041    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rf7m\" (UniqueName: \"kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.602078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.602137    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.602078 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.602253    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.703698 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.703665    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rf7m\" (UniqueName: \"kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.703929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.703769    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.704017 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.703949    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.704533 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.704504    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.706909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.706887    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.721226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.721191    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rf7m\" (UniqueName: \"kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m\") pod \"maas-api-revoke-sub-eb83ed34-r47lw\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:50.828344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:50.828314    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:51.143336 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:52:51.143302    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1399f95b_c193_4f1d_b5c3_38f19d40d1ab.slice/crio-dc45b37b055b57813115b57123bc7586bf6871b2f78adfe2d53b35af15825b6f WatchSource:0}: Error finding container dc45b37b055b57813115b57123bc7586bf6871b2f78adfe2d53b35af15825b6f: Status 404 returned error can't find the container with id dc45b37b055b57813115b57123bc7586bf6871b2f78adfe2d53b35af15825b6f
Oct 30 03:52:51.144456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:51.144407    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"]
Oct 30 03:52:51.475234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:51.475200    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw" event={"ID":"1399f95b-c193-4f1d-b5c3-38f19d40d1ab","Type":"ContainerStarted","Data":"dc45b37b055b57813115b57123bc7586bf6871b2f78adfe2d53b35af15825b6f"}
Oct 30 03:52:52.487962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:52.487857    4877 generic.go:356] "Generic (PLEG): container finished" podID="1399f95b-c193-4f1d-b5c3-38f19d40d1ab" containerID="70dd4dab51d0523ef5d87ae4c128856f6f2df6c909922780628326dc8daca419" exitCode=0
Oct 30 03:52:52.487962 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:52.487887    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw" event={"ID":"1399f95b-c193-4f1d-b5c3-38f19d40d1ab","Type":"ContainerDied","Data":"70dd4dab51d0523ef5d87ae4c128856f6f2df6c909922780628326dc8daca419"}
Oct 30 03:52:52.807649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:52.807621    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:52:52.816480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:52.816453    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:52:53.705570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.705542    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:53.764872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.764782    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m\" (UniqueName: \"kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m\") pod \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") "
Oct 30 03:52:53.764872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.764865    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca\") pod \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") "
Oct 30 03:52:53.765134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.764921    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca\") pod \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\" (UID: \"1399f95b-c193-4f1d-b5c3-38f19d40d1ab\") "
Oct 30 03:52:53.765362 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.765334    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca" pod "1399f95b-c193-4f1d-b5c3-38f19d40d1ab" (UID: "1399f95b-c193-4f1d-b5c3-38f19d40d1ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:52:53.765972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.765943    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:53.767286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.767260    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m" pod "1399f95b-c193-4f1d-b5c3-38f19d40d1ab" (UID: "1399f95b-c193-4f1d-b5c3-38f19d40d1ab"). InnerVolumeSpecName "kube-api-access-2rf7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:52:53.767774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.767748    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca" pod "1399f95b-c193-4f1d-b5c3-38f19d40d1ab" (UID: "1399f95b-c193-4f1d-b5c3-38f19d40d1ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:52:53.867149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.867117    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rf7m\" (UniqueName: \"kubernetes.io/projected/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-kube-api-access-2rf7m\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:53.867149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:53.867150    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1399f95b-c193-4f1d-b5c3-38f19d40d1ab-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:52:54.515607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:54.515575    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"
Oct 30 03:52:54.515791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:54.515600    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw" event={"ID":"1399f95b-c193-4f1d-b5c3-38f19d40d1ab","Type":"ContainerDied","Data":"dc45b37b055b57813115b57123bc7586bf6871b2f78adfe2d53b35af15825b6f"}
Oct 30 03:52:54.515791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:54.515685    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc45b37b055b57813115b57123bc7586bf6871b2f78adfe2d53b35af15825b6f"
Oct 30 03:52:55.800486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:55.800450    4877 scope.go:122] "RemoveContainer" containerID="be2d952a303709cb44422ebc6fa05edc9299cda1dbc484e942565f987c7cd976"
Oct 30 03:52:55.817476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:55.817450    4877 scope.go:122] "RemoveContainer" containerID="7f9e111aa21dda08c6e30e703cd1692cb0de77ac8e00989d663d5ecf5b52b137"
Oct 30 03:52:55.835589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:52:55.835028    4877 scope.go:122] "RemoveContainer" containerID="ed972cbd6621adc99efc4c8f44c8c9203650dbf633e717ec383f4d983a0471b9"
Oct 30 03:53:00.683088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.683052    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"]
Oct 30 03:53:00.683996 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.683300    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy" containerID="cri-o://1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa" gracePeriod=30
Oct 30 03:53:00.720211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.720180    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"]
Oct 30 03:53:00.721127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.721104    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1399f95b-c193-4f1d-b5c3-38f19d40d1ab" containerName="revoke-keys"
Oct 30 03:53:00.721127 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.721126    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1399f95b-c193-4f1d-b5c3-38f19d40d1ab" containerName="revoke-keys"
Oct 30 03:53:00.721338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.721323    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1399f95b-c193-4f1d-b5c3-38f19d40d1ab" containerName="revoke-keys"
Oct 30 03:53:00.727132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.727112    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.739246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.739219    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:53:00.739392 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.739315    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:53:00.742720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.742693    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"]
Oct 30 03:53:00.811733 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.811696    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.811903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.811754    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9qw\" (UniqueName: \"kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.812025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.811998    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.934496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.924627    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.934496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.924724    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9qw\" (UniqueName: \"kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.934496 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.924879    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.941461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.939348    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.941461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.940843    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:00.979968 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:00.979933    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9qw\" (UniqueName: \"kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw\") pod \"maas-api-revoke-sub-44943283-96szk\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:01.057040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:01.056999    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:01.422972 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:53:01.422931    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f231b8_3c9e_41d6_bd58_8cb578115096.slice/crio-f0efb3a201a960822c4ec64fe79d1258d90655aadb9e910c5475b2febb1d9a55 WatchSource:0}: Error finding container f0efb3a201a960822c4ec64fe79d1258d90655aadb9e910c5475b2febb1d9a55: Status 404 returned error can't find the container with id f0efb3a201a960822c4ec64fe79d1258d90655aadb9e910c5475b2febb1d9a55
Oct 30 03:53:01.426173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:01.426142    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"]
Oct 30 03:53:01.663890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:01.663837    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk" event={"ID":"86f231b8-3c9e-41d6-bd58-8cb578115096","Type":"ContainerStarted","Data":"f0efb3a201a960822c4ec64fe79d1258d90655aadb9e910c5475b2febb1d9a55"}
Oct 30 03:53:02.675924 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:02.675834    4877 generic.go:356] "Generic (PLEG): container finished" podID="86f231b8-3c9e-41d6-bd58-8cb578115096" containerID="1aac5d258011616bbf99a664dccb86871ba9e83c11ca981dc7d8ede481c14bd5" exitCode=0
Oct 30 03:53:02.676309 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:02.675918    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk" event={"ID":"86f231b8-3c9e-41d6-bd58-8cb578115096","Type":"ContainerDied","Data":"1aac5d258011616bbf99a664dccb86871ba9e83c11ca981dc7d8ede481c14bd5"}
Oct 30 03:53:03.889792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.889763    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:03.987531 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.987387    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw\" (UniqueName: \"kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw\") pod \"86f231b8-3c9e-41d6-bd58-8cb578115096\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") "
Oct 30 03:53:03.987758 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.987733    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca\") pod \"86f231b8-3c9e-41d6-bd58-8cb578115096\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") "
Oct 30 03:53:03.987856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.987833    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca\") pod \"86f231b8-3c9e-41d6-bd58-8cb578115096\" (UID: \"86f231b8-3c9e-41d6-bd58-8cb578115096\") "
Oct 30 03:53:03.988291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.988258    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca" pod "86f231b8-3c9e-41d6-bd58-8cb578115096" (UID: "86f231b8-3c9e-41d6-bd58-8cb578115096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:03.988628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.988605    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:03.989798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.989773    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw" pod "86f231b8-3c9e-41d6-bd58-8cb578115096" (UID: "86f231b8-3c9e-41d6-bd58-8cb578115096"). InnerVolumeSpecName "kube-api-access-xk9qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:03.990343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:03.990320    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca" pod "86f231b8-3c9e-41d6-bd58-8cb578115096" (UID: "86f231b8-3c9e-41d6-bd58-8cb578115096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:53:04.089949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:04.089910    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f231b8-3c9e-41d6-bd58-8cb578115096-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:04.089949 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:04.089943    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9qw\" (UniqueName: \"kubernetes.io/projected/86f231b8-3c9e-41d6-bd58-8cb578115096-kube-api-access-xk9qw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:04.696348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:04.696311    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk" event={"ID":"86f231b8-3c9e-41d6-bd58-8cb578115096","Type":"ContainerDied","Data":"f0efb3a201a960822c4ec64fe79d1258d90655aadb9e910c5475b2febb1d9a55"}
Oct 30 03:53:04.696348 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:04.696329    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"
Oct 30 03:53:04.696616 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:04.696358    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0efb3a201a960822c4ec64fe79d1258d90655aadb9e910c5475b2febb1d9a55"
Oct 30 03:53:06.050028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.049993    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:53:06.134169 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134138    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134187    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134290    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134464    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134500    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:06.134638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134525    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134629    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134648    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:06.134763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134676    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134789    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.134875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134855    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:06.134999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.134928    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket\") pod \"ed33a285-7366-4359-93ff-1d4118773ab3\" (UID: \"ed33a285-7366-4359-93ff-1d4118773ab3\") "
Oct 30 03:53:06.135288 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.135264    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:06.135604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.135577    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:06.136109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.136086    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed33a285-7366-4359-93ff-1d4118773ab3-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.136243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.136229    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.136390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.136373    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.136505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.136491    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.136602 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.136589    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.137815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.137789    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:53:06.138025 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.137992    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:06.138135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.138089    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:06.138844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.138822    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89" pod "ed33a285-7366-4359-93ff-1d4118773ab3" (UID: "ed33a285-7366-4359-93ff-1d4118773ab3"). InnerVolumeSpecName "kube-api-access-mqn89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:06.238236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.238144    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqn89\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-kube-api-access-mqn89\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.238236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.238179    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed33a285-7366-4359-93ff-1d4118773ab3-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.238236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.238203    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed33a285-7366-4359-93ff-1d4118773ab3-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.238236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.238211    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed33a285-7366-4359-93ff-1d4118773ab3-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:06.723488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.723399    4877 generic.go:356] "Generic (PLEG): container finished" podID="ed33a285-7366-4359-93ff-1d4118773ab3" containerID="1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa" exitCode=0
Oct 30 03:53:06.723488 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.723489    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"
Oct 30 03:53:06.723765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.723499    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" event={"ID":"ed33a285-7366-4359-93ff-1d4118773ab3","Type":"ContainerDied","Data":"1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa"}
Oct 30 03:53:06.723765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.723560    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw" event={"ID":"ed33a285-7366-4359-93ff-1d4118773ab3","Type":"ContainerDied","Data":"dacce38b1e1e5cf644ac773426c9453835913c77ba311986ce5804cbde88bae5"}
Oct 30 03:53:06.723765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.723585    4877 scope.go:122] "RemoveContainer" containerID="1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa"
Oct 30 03:53:06.743132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.743089    4877 scope.go:122] "RemoveContainer" containerID="1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"
Oct 30 03:53:06.755375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.755347    4877 scope.go:122] "RemoveContainer" containerID="1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa"
Oct 30 03:53:06.755740 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:06.755714    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa\": container with ID starting with 1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa not found: ID does not exist" containerID="1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa"
Oct 30 03:53:06.755858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.755749    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa"} err="failed to get container status \"1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa\": rpc error: code = NotFound desc = could not find container \"1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa\": container with ID starting with 1ba9389f75e3714c544f20d0ba8310f1d6edd1415022212c8f9dbc9259d43caa not found: ID does not exist"
Oct 30 03:53:06.755858 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.755770    4877 scope.go:122] "RemoveContainer" containerID="1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"
Oct 30 03:53:06.756117 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:06.756095    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277\": container with ID starting with 1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277 not found: ID does not exist" containerID="1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"
Oct 30 03:53:06.756181 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.756126    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277"} err="failed to get container status \"1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277\": rpc error: code = NotFound desc = could not find container \"1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277\": container with ID starting with 1125c1ee415d1dd798e52ca69ec6034769cba3310b9f5a66bf5d0a3bf7032277 not found: ID does not exist"
Oct 30 03:53:06.780605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.780570    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"]
Oct 30 03:53:06.790159 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:06.790126    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-p65hw"]
Oct 30 03:53:07.705047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:07.705008    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" path="/var/lib/kubelet/pods/ed33a285-7366-4359-93ff-1d4118773ab3/volumes"
Oct 30 03:53:13.929411 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.929360    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"]
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930613    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy"
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930634    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy"
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930655    4877 cpu_manager.go:414] "containerMap: removing container" podUID="86f231b8-3c9e-41d6-bd58-8cb578115096" containerName="revoke-keys"
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930661    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f231b8-3c9e-41d6-bd58-8cb578115096" containerName="revoke-keys"
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930793    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy"
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930804    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy"
Oct 30 03:53:13.931625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.930810    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f231b8-3c9e-41d6-bd58-8cb578115096" containerName="revoke-keys"
Oct 30 03:53:13.935036 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.934967    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:13.949028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.949006    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"]
Oct 30 03:53:13.951307 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.950604    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\""
Oct 30 03:53:13.951703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.951533    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\""
Oct 30 03:53:13.954701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.951884    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-df757712a33a\""
Oct 30 03:53:13.954701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.952135    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\""
Oct 30 03:53:13.957449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.957325    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\""
Oct 30 03:53:13.961448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.960644    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-mpmb2\""
Oct 30 03:53:13.963589 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.963558    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:13.963697 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.963611    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kbc\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:13.964258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.963807    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:13.964258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:13.963851    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.064999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.064969    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.065171 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.065006    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kbc\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.065259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.065195    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.065259 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.065234    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.065849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.065817    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.066028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.066001    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.068136 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.068111    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.082185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.082159    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kbc\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc\") pod \"payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.257905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.257821    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:14.573234 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:53:14.573202    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216bca3f_5cfb_4fb8_ad32_a885e16008ee.slice/crio-e706c1fedd416d526a897eb8002017319480d6fa4b8990b79fbdfc94f1ce9393 WatchSource:0}: Error finding container e706c1fedd416d526a897eb8002017319480d6fa4b8990b79fbdfc94f1ce9393: Status 404 returned error can't find the container with id e706c1fedd416d526a897eb8002017319480d6fa4b8990b79fbdfc94f1ce9393
Oct 30 03:53:14.575094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.575072    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"]
Oct 30 03:53:14.939023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.938922    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" event={"ID":"216bca3f-5cfb-4fb8-ad32-a885e16008ee","Type":"ContainerStarted","Data":"458af8a01da60a8d4baec4ab0a66f26b9d5848249154131803e5195c791b8acb"}
Oct 30 03:53:14.939977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.939947    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" event={"ID":"216bca3f-5cfb-4fb8-ad32-a885e16008ee","Type":"ContainerStarted","Data":"e706c1fedd416d526a897eb8002017319480d6fa4b8990b79fbdfc94f1ce9393"}
Oct 30 03:53:14.956445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:14.941327    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:15.115645 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:15.115140    4877 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-df757712a33a: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:15.115645 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:15.115171    4877 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:15.115645 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:15.115250    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:15.615220511 +0000 UTC m=+3024.044109921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : secret "agc-cred-df757712a33a" not found
Oct 30 03:53:15.425741 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:15.425678    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" podStartSLOduration=2.425660171 podStartE2EDuration="2.425660171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:15.034704891 +0000 UTC m=+3023.463594308" watchObservedRunningTime="2026-10-30 03:53:15.425660171 +0000 UTC m=+3023.854549590"
Oct 30 03:53:15.430539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:15.430507    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"]
Oct 30 03:53:15.623713 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:15.623669    4877 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-df757712a33a: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:15.623713 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:15.623701    4877 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:15.623935 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:15.623770    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:16.623751183 +0000 UTC m=+3025.052640599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : secret "agc-cred-df757712a33a" not found
Oct 30 03:53:16.018446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:16.018252    4877 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" secret="" err="secret \"payload-processing-external-model-e2e--2cc204ae-dockercfg-mpmb2\" not found"
Oct 30 03:53:16.146326 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.146292    4877 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-plugins-e2e-extmod-b-372480: configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:16.146557 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.146372    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:16.646351629 +0000 UTC m=+3025.075241045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:16.229884 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.229846    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:53:16.655452 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.655380    4877 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-plugins-e2e-extmod-b-372480: configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:16.655622 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.655503    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:17.655482102 +0000 UTC m=+3026.084371520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:16.655622 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.655395    4877 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-df757712a33a: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:16.655622 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.655530    4877 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:16.655622 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:16.655567    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:18.655555768 +0000 UTC m=+3027.084445180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : secret "agc-cred-df757712a33a" not found
Oct 30 03:53:17.674322 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:17.674293    4877 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-plugins-e2e-extmod-b-372480: configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:17.674731 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:17.674363    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:19.674348288 +0000 UTC m=+3028.103237684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:18.692477 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:18.692440    4877 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-df757712a33a: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:18.692477 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:18.692469    4877 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:18.693015 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:18.692530    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:22.692512226 +0000 UTC m=+3031.121401623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : secret "agc-cred-df757712a33a" not found
Oct 30 03:53:19.709963 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:19.709928    4877 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-plugins-e2e-extmod-b-372480: configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:19.710476 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:19.710013    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:23.709992991 +0000 UTC m=+3032.138882404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-372480" not found
Oct 30 03:53:22.039450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:22.035556    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" podUID="216bca3f-5cfb-4fb8-ad32-a885e16008ee" containerName="payload-processing" containerID="cri-o://458af8a01da60a8d4baec4ab0a66f26b9d5848249154131803e5195c791b8acb" gracePeriod=25
Oct 30 03:53:22.759448 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:22.756224    4877 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-372480/agc-cred-df757712a33a: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:22.759448 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:22.756260    4877 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2: secret "agc-cred-df757712a33a" not found
Oct 30 03:53:22.759448 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:22.756328    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials podName:216bca3f-5cfb-4fb8-ad32-a885e16008ee nodeName:}" failed. No retries permitted until 2026-10-30 03:53:30.756308878 +0000 UTC m=+3039.185198299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee") : secret "agc-cred-df757712a33a" not found
Oct 30 03:53:22.955053 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:22.955003    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Oct 30 03:53:23.082262 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.082208    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 03:53:23.212058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:22.943610    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded"
Oct 30 03:53:23.230397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:22.943669    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:53:23.248925 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.248887    4877 generic.go:356] "Generic (PLEG): container finished" podID="216bca3f-5cfb-4fb8-ad32-a885e16008ee" containerID="458af8a01da60a8d4baec4ab0a66f26b9d5848249154131803e5195c791b8acb" exitCode=0
Oct 30 03:53:23.249122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.248961    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" event={"ID":"216bca3f-5cfb-4fb8-ad32-a885e16008ee","Type":"ContainerDied","Data":"458af8a01da60a8d4baec4ab0a66f26b9d5848249154131803e5195c791b8acb"}
Oct 30 03:53:23.407452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.402133    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:23.424863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.424835    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume\") pod \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") "
Oct 30 03:53:23.425124 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.425110    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume\") pod \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") "
Oct 30 03:53:23.425281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.425270    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials\") pod \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") "
Oct 30 03:53:23.425464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.425444    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc\") pod \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\" (UID: \"216bca3f-5cfb-4fb8-ad32-a885e16008ee\") "
Oct 30 03:53:23.425746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.425709    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume" pod "216bca3f-5cfb-4fb8-ad32-a885e16008ee" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:23.425977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.425955    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume" pod "216bca3f-5cfb-4fb8-ad32-a885e16008ee" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:23.426525 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.426504    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:23.426621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.426612    4877 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/216bca3f-5cfb-4fb8-ad32-a885e16008ee-routing-overlay-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:23.428029 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.427997    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc" pod "216bca3f-5cfb-4fb8-ad32-a885e16008ee" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee"). InnerVolumeSpecName "kube-api-access-z7kbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:23.428412 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.428386    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials" pod "216bca3f-5cfb-4fb8-ad32-a885e16008ee" (UID: "216bca3f-5cfb-4fb8-ad32-a885e16008ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:23.530802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.529869    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7kbc\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-kube-api-access-z7kbc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:23.530802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:23.529911    4877 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/216bca3f-5cfb-4fb8-ad32-a885e16008ee-provider-credentials\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:53:24.265397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:24.265296    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2" event={"ID":"216bca3f-5cfb-4fb8-ad32-a885e16008ee","Type":"ContainerDied","Data":"e706c1fedd416d526a897eb8002017319480d6fa4b8990b79fbdfc94f1ce9393"}
Oct 30 03:53:24.265397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:24.265374    4877 scope.go:122] "RemoveContainer" containerID="458af8a01da60a8d4baec4ab0a66f26b9d5848249154131803e5195c791b8acb"
Oct 30 03:53:24.265941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:24.265309    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"
Oct 30 03:53:24.364021 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:24.363982    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"]
Oct 30 03:53:24.373413 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:24.373387    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-d9b75wpn2"]
Oct 30 03:53:25.749994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:25.749956    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216bca3f-5cfb-4fb8-ad32-a885e16008ee" path="/var/lib/kubelet/pods/216bca3f-5cfb-4fb8-ad32-a885e16008ee/volumes"
Oct 30 03:53:26.151372 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.151337    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"]
Oct 30 03:53:26.152876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.152847    4877 cpu_manager.go:414] "containerMap: removing container" podUID="216bca3f-5cfb-4fb8-ad32-a885e16008ee" containerName="payload-processing"
Oct 30 03:53:26.153052 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.153038    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216bca3f-5cfb-4fb8-ad32-a885e16008ee" containerName="payload-processing"
Oct 30 03:53:26.153151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.153141    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy"
Oct 30 03:53:26.153234 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.153224    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed33a285-7366-4359-93ff-1d4118773ab3" containerName="istio-proxy"
Oct 30 03:53:26.153576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.153559    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="216bca3f-5cfb-4fb8-ad32-a885e16008ee" containerName="payload-processing"
Oct 30 03:53:26.162672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.162652    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.166174 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.166149    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"]
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.184400    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4206d9-openshift-default-dockercfg-6qz7k\""
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275490    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275569    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275631    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275661    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275698    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275765    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275906    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.275996    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.278655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.276028    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksgm8\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419162    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419267    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419307    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksgm8\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419519    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419582    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419641    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419671    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419704    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.440520 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.419774    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.468447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.464457    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.468638 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.468612    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.484593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.474583    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.484593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.478828    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.515450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.511288    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.635019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.634627    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksgm8\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.701246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.684046    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.701246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.696649    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.747591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.727296    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy\") pod \"e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") " pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:26.910449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:26.909723    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:27.349726 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:53:27.349688    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636f9bc8_0c75_4a86_8a8e_67d950e052d0.slice/crio-8dcf6813d5b36855763b24d69ffb9fbf3a3eedc1843e426e4af5b79f3da79a97 WatchSource:0}: Error finding container 8dcf6813d5b36855763b24d69ffb9fbf3a3eedc1843e426e4af5b79f3da79a97: Status 404 returned error can't find the container with id 8dcf6813d5b36855763b24d69ffb9fbf3a3eedc1843e426e4af5b79f3da79a97
Oct 30 03:53:27.351019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:27.350993    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"]
Oct 30 03:53:27.352937 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:27.352902    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:27.353047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:27.352974    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:27.353047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:27.353004    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:27.474005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:27.473970    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" event={"ID":"636f9bc8-0c75-4a86-8a8e-67d950e052d0","Type":"ContainerStarted","Data":"8dcf6813d5b36855763b24d69ffb9fbf3a3eedc1843e426e4af5b79f3da79a97"}
Oct 30 03:53:28.576468 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:28.576392    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" event={"ID":"636f9bc8-0c75-4a86-8a8e-67d950e052d0","Type":"ContainerStarted","Data":"8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"}
Oct 30 03:53:28.741461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:28.738039    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podStartSLOduration=2.738018038 podStartE2EDuration="2.738018038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:28.676391096 +0000 UTC m=+3037.105280515" watchObservedRunningTime="2026-10-30 03:53:28.738018038 +0000 UTC m=+3037.166907457"
Oct 30 03:53:28.912967 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:28.912873    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:28.924077 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:28.924047    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:29.621765 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:29.621725    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:29.623108 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:29.623082    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:53:32.221599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:32.221560    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:53:32.222109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:32.221625    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:53:40.342614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.342583    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"]
Oct 30 03:53:40.359027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.358962    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.379335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.378965    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"]
Oct 30 03:53:40.493114 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493073    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493122    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493199    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493282    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493373 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493316    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493434    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhvq\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493500    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493526    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.493701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.493626    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595351    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhvq\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595467    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595506    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595586    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595780    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595822    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.595908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595859    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595906    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595939    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596071 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.595983    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.596297    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.596466    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596599 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.596521    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.596722 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.596702    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.609652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.609619    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.609806 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.609757    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.612773 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.612752    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhvq\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.613569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.613551    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.721862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.721593    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:40.836913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.836881    4877 generic.go:356] "Generic (PLEG): container finished" podID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerID="d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4" exitCode=0
Oct 30 03:53:40.837190 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.837162    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" event={"ID":"a633a9a2-e8ac-415b-95c3-4d478846e0ce","Type":"ContainerDied","Data":"d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"}
Oct 30 03:53:40.837630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.837608    4877 scope.go:122] "RemoveContainer" containerID="d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"
Oct 30 03:53:40.840715 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.840677    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:40.840848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.840769    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:40.840917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:40.840883    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.224160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.224120    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"]
Oct 30 03:53:41.225072 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:53:41.225023    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a85e54_af6e_4b2f_8619_a69e45d9da7c.slice/crio-0a358f139f1b1ba99e22eebdbfb976b22ef13d5e78a018137052b394ca561c3e WatchSource:0}: Error finding container 0a358f139f1b1ba99e22eebdbfb976b22ef13d5e78a018137052b394ca561c3e: Status 404 returned error can't find the container with id 0a358f139f1b1ba99e22eebdbfb976b22ef13d5e78a018137052b394ca561c3e
Oct 30 03:53:41.229315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.229281    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.229442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.229364    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.229442 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.229395    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.851727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.851681    4877 generic.go:356] "Generic (PLEG): container finished" podID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerID="fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd" exitCode=0
Oct 30 03:53:41.852274 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.852011    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" event={"ID":"d276f194-aaa3-4df4-8104-e8844bb5aa82","Type":"ContainerDied","Data":"fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"}
Oct 30 03:53:41.852712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.852690    4877 scope.go:122] "RemoveContainer" containerID="fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"
Oct 30 03:53:41.856315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.856281    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.856479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.856374    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.856479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.856439    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:53:41.858170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.857640    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" event={"ID":"73a85e54-af6e-4b2f-8619-a69e45d9da7c","Type":"ContainerStarted","Data":"ef4f5549cde91989d463efc9b556f51661a630d4786afdbb8cf085630895a928"}
Oct 30 03:53:41.858170 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.857738    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" event={"ID":"73a85e54-af6e-4b2f-8619-a69e45d9da7c","Type":"ContainerStarted","Data":"0a358f139f1b1ba99e22eebdbfb976b22ef13d5e78a018137052b394ca561c3e"}
Oct 30 03:53:41.860675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:41.860650    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" event={"ID":"a633a9a2-e8ac-415b-95c3-4d478846e0ce","Type":"ContainerStarted","Data":"b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056"}
Oct 30 03:53:42.058709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.058647    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podStartSLOduration=2.05862783 podStartE2EDuration="2.05862783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:42.057012073 +0000 UTC m=+3050.485901492" watchObservedRunningTime="2026-10-30 03:53:42.05862783 +0000 UTC m=+3050.487517249"
Oct 30 03:53:42.689576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.689540    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:53:42.701450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.697003    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 03:53:42.701450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.697059    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 03:53:42.729534 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.729501    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:42.729903 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.729877    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 03:53:42.730075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.730051    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 03:53:42.874841 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:42.874801    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" event={"ID":"d276f194-aaa3-4df4-8104-e8844bb5aa82","Type":"ContainerStarted","Data":"97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c"}
Oct 30 03:53:43.647449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.643537    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:53:43.698450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.686817    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 03:53:43.698450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.686898    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 03:53:43.739456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.734450    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 03:53:43.739456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.734511    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 03:53:43.854448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.853786    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:53:43.861465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.859298    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 03:53:43.861465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:43.859360    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 03:53:44.729361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:44.728564    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 03:53:44.729361 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:44.728651    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 03:53:44.763337 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:44.763299    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 03:53:44.766677 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:44.766636    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 03:53:44.857449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:44.857078    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 03:53:44.857449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:44.857136    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 03:53:45.703447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:45.699702    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 03:53:45.703447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:45.699762    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 03:53:45.764490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:45.763085    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 03:53:45.764490 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:45.763153    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 03:53:45.861465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:45.849439    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 03:53:45.861780 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:45.861745    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 03:53:46.200459 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.180692    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:53:46.364314 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:53:46.364280    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:53:46.688483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.687253    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 03:53:46.688483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.687349    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 03:53:46.741448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.740574    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 03:53:46.741448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.740646    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 03:53:46.939219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.939119    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 03:53:46.977456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:46.977391    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 03:53:47.688876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:47.688821    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Oct 30 03:53:47.689223 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:47.689189    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Oct 30 03:53:47.722047 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:47.722013    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 30 03:53:47.722315 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:47.722290    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 30 03:53:47.847090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:47.847051    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 03:53:47.848376 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:47.848329    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 03:53:48.706219 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.706182    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:53:48.707844 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.707816    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 03:53:48.727612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.727585    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:48.727959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.727935    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:48.728630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.728609    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 03:53:48.847168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.847131    4877 patch_prober.go:29] interesting pod/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 03:53:48.847366 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:48.847192    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 03:53:49.852384 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:49.852349    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:53:49.853772 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:49.853741    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 03:53:55.899710 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:55.899678    4877 scope.go:122] "RemoveContainer" containerID="b126087b05a153736f1dbf19c4f4bc59dfd8c0ef91795c65e88f658fcaeb82a1"
Oct 30 03:53:55.914929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:55.914881    4877 scope.go:122] "RemoveContainer" containerID="9fa77fcc265ed130e058497f025dcc5b21fc98a774158b315ba21a184da0d735"
Oct 30 03:53:55.927559 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:53:55.927531    4877 scope.go:122] "RemoveContainer" containerID="a9f337bb65ad7a3a1f5dcbf10d799fe6c04f515b88ec55ba9d86a566b4abec64"
Oct 30 03:54:07.032822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:07.032788    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"]
Oct 30 03:54:07.040365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:07.040337    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4725d63-xbchw"]
Oct 30 03:54:07.701584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:07.701548    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af601240-2c67-4ab7-8a64-17d6f390dabb" path="/var/lib/kubelet/pods/af601240-2c67-4ab7-8a64-17d6f390dabb/volumes"
Oct 30 03:54:10.430370 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.430334    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"]
Oct 30 03:54:10.437585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.437560    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.463859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.463825    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"]
Oct 30 03:54:10.477812 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.477763    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.477955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.477826    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.477955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.477937    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.478045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.478007    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.478045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.478037    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.478160 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.478144    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.478207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.478180    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.478265 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.478247    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfx9\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.478483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.478458    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.579652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.579618    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.579652 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.579655    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.579923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.579705    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfx9\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.579923 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.579843    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.579936    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.579983    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580081    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580160    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580216 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580191    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580225    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580280    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580647 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580624    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580643    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.580720 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.580685    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.582195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.582172    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.582692 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.582673    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.596393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.596366    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.596778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.596758    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfx9\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:10.760277 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:10.760194    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:11.088481 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:54:11.088443    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8df1dd_105c_4207_bcaa_596b7f0351e1.slice/crio-862755d1aee842f19e13528850d3fdb0400584399643c299bad71a082ad063de WatchSource:0}: Error finding container 862755d1aee842f19e13528850d3fdb0400584399643c299bad71a082ad063de: Status 404 returned error can't find the container with id 862755d1aee842f19e13528850d3fdb0400584399643c299bad71a082ad063de
Oct 30 03:54:11.089529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.089495    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"]
Oct 30 03:54:11.092549 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.092506    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:54:11.092695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.092595    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:54:11.092695 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.092642    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:54:11.669592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.669078    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" event={"ID":"de8df1dd-105c-4207-bcaa-596b7f0351e1","Type":"ContainerStarted","Data":"0cf554f7f48a9fda54b22251074b869fcd484c17a81f8a00572611066d20e905"}
Oct 30 03:54:11.669592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.669151    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" event={"ID":"de8df1dd-105c-4207-bcaa-596b7f0351e1","Type":"ContainerStarted","Data":"862755d1aee842f19e13528850d3fdb0400584399643c299bad71a082ad063de"}
Oct 30 03:54:11.733619 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.733557    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podStartSLOduration=1.733542698 podStartE2EDuration="1.733542698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:11.717385006 +0000 UTC m=+3080.146274448" watchObservedRunningTime="2026-10-30 03:54:11.733542698 +0000 UTC m=+3080.162432115"
Oct 30 03:54:11.761243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:11.761209    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:12.762203 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:12.762168    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:54:12.762819 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:12.762227    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 03:54:13.761545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:13.761514    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:54:13.761728 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:13.761562    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 03:54:14.760887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:14.760856    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:54:14.761388 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:14.760921    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 03:54:15.760675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:15.760644    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:54:15.760879 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:15.760694    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 03:54:16.238219 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:54:16.238178    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:54:16.761041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:16.761005    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Oct 30 03:54:16.761220 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:16.761066    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Oct 30 03:54:17.766403 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:17.766369    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:17.766887 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:17.766869    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:17.767560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:17.767540    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:54:18.028778 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:18.028742    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"]
Oct 30 03:54:18.034032 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:18.034000    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37628bc8-zpb7d"]
Oct 30 03:54:19.702519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:19.702483    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29acc96-fa42-4214-bbb8-9e33adc52a2d" path="/var/lib/kubelet/pods/e29acc96-fa42-4214-bbb8-9e33adc52a2d/volumes"
Oct 30 03:54:25.555462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.551480    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"]
Oct 30 03:54:25.557753 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.557722    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.577447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.562030    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"]
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.674703    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.674805    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.674866    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm786\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.674907    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.675020    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.675070    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.675091    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.675224    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.691445 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.675299    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.814608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.814724    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.814850    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.815528    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.815946    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.816204    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.816283    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm786\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.816337    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.816440    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.816525    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.817505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.816553    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.821212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.818735    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.821212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.819730    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.902522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.901291    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.916452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.906312    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.916452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.907230    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm786\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.916452 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.908655    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:25.953448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:25.943916    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:26.050566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:26.046676    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:26.556592 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:54:26.556563    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9a8a23_ed54_462d_bafa_2aa064a9d7c1.slice/crio-8cc8a2e13de5424de0d8ea28a97d034e65abf092c25d4160333d9a5652f3be88 WatchSource:0}: Error finding container 8cc8a2e13de5424de0d8ea28a97d034e65abf092c25d4160333d9a5652f3be88: Status 404 returned error can't find the container with id 8cc8a2e13de5424de0d8ea28a97d034e65abf092c25d4160333d9a5652f3be88
Oct 30 03:54:26.557703 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:26.557673    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"]
Oct 30 03:54:26.560669 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:26.560631    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:54:26.560752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:26.560720    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:54:26.560799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:26.560784    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:54:27.000147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:27.000045    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" event={"ID":"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1","Type":"ContainerStarted","Data":"82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4"}
Oct 30 03:54:27.000147 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:27.000111    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" event={"ID":"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1","Type":"ContainerStarted","Data":"8cc8a2e13de5424de0d8ea28a97d034e65abf092c25d4160333d9a5652f3be88"}
Oct 30 03:54:27.042824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:27.042768    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podStartSLOduration=2.042753481 podStartE2EDuration="2.042753481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:27.037870957 +0000 UTC m=+3095.466760377" watchObservedRunningTime="2026-10-30 03:54:27.042753481 +0000 UTC m=+3095.471642899"
Oct 30 03:54:27.049319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:27.049086    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:54:27.049319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:27.049139    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 03:54:27.049574 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:27.049374    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:28.047664 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:28.047627    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:54:28.048172 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:28.047700    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 03:54:29.047657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:29.047619    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:54:29.047849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:29.047673    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 03:54:30.047610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:30.047577    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:54:30.047813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:30.047627    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 03:54:31.047956 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:31.047922    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:54:31.048536 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:31.047983    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 03:54:32.047393 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:32.047356    4877 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Oct 30 03:54:32.047661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:32.047440    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Oct 30 03:54:33.085212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:33.085178    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:33.086135 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:33.086107    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:33.087714 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:33.087693    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:54:46.234397 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:54:46.234309    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:54:53.573518 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.573476    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"]
Oct 30 03:54:53.578483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.578461    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.596905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.596878    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"]
Oct 30 03:54:53.599276 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.599252    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:54:53.599474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.599419    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:54:53.720541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.720500    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsf2n\" (UniqueName: \"kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.720730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.720691    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.720861 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.720838    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.823046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.823006    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsf2n\" (UniqueName: \"kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.823235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.823161    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.823320 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.823282    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.824205 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.824134    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.825855 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.825832    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.840204 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.840175    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsf2n\" (UniqueName: \"kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n\") pod \"maas-api-revoke-sub-18710234-v6qwg\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:53.895495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:53.895400    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:54.220060 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:54.220032    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"]
Oct 30 03:54:54.223986 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:54:54.223946    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec3120d_e1f0_4cff_a33b_3b1c11050e6e.slice/crio-c857d2939979f3242f9e5d154927542ea2db5293ead7346a917f0f3ccd6388e4 WatchSource:0}: Error finding container c857d2939979f3242f9e5d154927542ea2db5293ead7346a917f0f3ccd6388e4: Status 404 returned error can't find the container with id c857d2939979f3242f9e5d154927542ea2db5293ead7346a917f0f3ccd6388e4
Oct 30 03:54:54.468267 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:54.468225    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg" event={"ID":"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e","Type":"ContainerStarted","Data":"c857d2939979f3242f9e5d154927542ea2db5293ead7346a917f0f3ccd6388e4"}
Oct 30 03:54:55.480080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:55.480042    4877 generic.go:356] "Generic (PLEG): container finished" podID="3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" containerID="b258dcbd506fc4328a6002557a359e1e4198e5afe3365458c0924eb77fc6e040" exitCode=0
Oct 30 03:54:55.480592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:55.480148    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg" event={"ID":"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e","Type":"ContainerDied","Data":"b258dcbd506fc4328a6002557a359e1e4198e5afe3365458c0924eb77fc6e040"}
Oct 30 03:54:56.687661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.687636    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:56.775513 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.775480    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n\" (UniqueName: \"kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n\") pod \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") "
Oct 30 03:54:56.775729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.775610    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca\") pod \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") "
Oct 30 03:54:56.775729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.775705    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca\") pod \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\" (UID: \"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e\") "
Oct 30 03:54:56.776182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.776147    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca" pod "3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" (UID: "3ec3120d-e1f0-4cff-a33b-3b1c11050e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:54:56.776654 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.776627    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:54:56.778253 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.778219    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca" pod "3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" (UID: "3ec3120d-e1f0-4cff-a33b-3b1c11050e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:54:56.778668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.778641    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n" pod "3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" (UID: "3ec3120d-e1f0-4cff-a33b-3b1c11050e6e"). InnerVolumeSpecName "kube-api-access-jsf2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:54:56.878441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.878397    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsf2n\" (UniqueName: \"kubernetes.io/projected/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-kube-api-access-jsf2n\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:54:56.878441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:56.878444    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:54:57.505260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:57.505231    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"
Oct 30 03:54:57.505260 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:57.505247    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg" event={"ID":"3ec3120d-e1f0-4cff-a33b-3b1c11050e6e","Type":"ContainerDied","Data":"c857d2939979f3242f9e5d154927542ea2db5293ead7346a917f0f3ccd6388e4"}
Oct 30 03:54:57.505551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:54:57.505304    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c857d2939979f3242f9e5d154927542ea2db5293ead7346a917f0f3ccd6388e4"
Oct 30 03:55:03.490449 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:03.485066    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:55:03.868735 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:03.868686    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:03.875447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:03.869053    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:03.940564 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:03.940511    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:04.094561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:04.094512    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:04.277749 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:04.277705    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:04.574656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:04.574554    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:04.575506 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:04.575469    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:55:04.998623 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:04.995587    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 03:55:06.434075 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.434038    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"]
Oct 30 03:55:06.435529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.435499    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" containerName="revoke-keys"
Oct 30 03:55:06.435529 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.435525    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" containerName="revoke-keys"
Oct 30 03:55:06.435763 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.435750    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" containerName="revoke-keys"
Oct 30 03:55:06.441954 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.441927    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"]
Oct 30 03:55:06.442113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.442059    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.464726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.464694    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:55:06.464726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.464700    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:55:06.536481 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.536447    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.536649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.536632    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4795f\" (UniqueName: \"kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.537005 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.536969    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.638103 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.638061    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4795f\" (UniqueName: \"kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.638334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.638286    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.638334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.638323    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.639335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.639300    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.640786 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.640767    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.655001 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.654969    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4795f\" (UniqueName: \"kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f\") pod \"maas-api-revoke-sub-9d7c8396-l9vrz\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:06.766838 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:06.766739    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:07.124439 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:07.124320    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"]
Oct 30 03:55:08.163856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:08.163816    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz" event={"ID":"781be59f-c261-49e7-a461-ff1ba0fa0c72","Type":"ContainerStarted","Data":"290342b791911f931e04112116a391047a7f23e6f0d186568c807792ba129dfe"}
Oct 30 03:55:08.164474 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:08.164439    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz" event={"ID":"781be59f-c261-49e7-a461-ff1ba0fa0c72","Type":"ContainerStarted","Data":"fec759a6d5bdc52c55a0d6f5220eed0972b998fbfd2ee7f2a47479dfab794fd6"}
Oct 30 03:55:08.202212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:08.202135    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz" podStartSLOduration=2.202098922 podStartE2EDuration="2.202098922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:08.201130043 +0000 UTC m=+3136.630019462" watchObservedRunningTime="2026-10-30 03:55:08.202098922 +0000 UTC m=+3136.630988340"
Oct 30 03:55:09.176056 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:09.176017    4877 generic.go:356] "Generic (PLEG): container finished" podID="781be59f-c261-49e7-a461-ff1ba0fa0c72" containerID="290342b791911f931e04112116a391047a7f23e6f0d186568c807792ba129dfe" exitCode=0
Oct 30 03:55:09.176510 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:09.176057    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz" event={"ID":"781be59f-c261-49e7-a461-ff1ba0fa0c72","Type":"ContainerDied","Data":"290342b791911f931e04112116a391047a7f23e6f0d186568c807792ba129dfe"}
Oct 30 03:55:10.514303 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.514226    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:10.554177 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.554142    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f\" (UniqueName: \"kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f\") pod \"781be59f-c261-49e7-a461-ff1ba0fa0c72\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") "
Oct 30 03:55:10.554353 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.554271    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca\") pod \"781be59f-c261-49e7-a461-ff1ba0fa0c72\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") "
Oct 30 03:55:10.554472 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.554360    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca\") pod \"781be59f-c261-49e7-a461-ff1ba0fa0c72\" (UID: \"781be59f-c261-49e7-a461-ff1ba0fa0c72\") "
Oct 30 03:55:10.554689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.554665    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca" pod "781be59f-c261-49e7-a461-ff1ba0fa0c72" (UID: "781be59f-c261-49e7-a461-ff1ba0fa0c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:10.555189 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.555165    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:10.556736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.556706    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f" pod "781be59f-c261-49e7-a461-ff1ba0fa0c72" (UID: "781be59f-c261-49e7-a461-ff1ba0fa0c72"). InnerVolumeSpecName "kube-api-access-4795f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:10.556839 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.556815    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca" pod "781be59f-c261-49e7-a461-ff1ba0fa0c72" (UID: "781be59f-c261-49e7-a461-ff1ba0fa0c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:55:10.656672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.656639    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/781be59f-c261-49e7-a461-ff1ba0fa0c72-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:10.656672 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.656668    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4795f\" (UniqueName: \"kubernetes.io/projected/781be59f-c261-49e7-a461-ff1ba0fa0c72-kube-api-access-4795f\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:10.898539 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.898492    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"]
Oct 30 03:55:10.900460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.900405    4877 cpu_manager.go:414] "containerMap: removing container" podUID="781be59f-c261-49e7-a461-ff1ba0fa0c72" containerName="revoke-keys"
Oct 30 03:55:10.900680 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.900664    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="781be59f-c261-49e7-a461-ff1ba0fa0c72" containerName="revoke-keys"
Oct 30 03:55:10.901045 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.901028    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="781be59f-c261-49e7-a461-ff1ba0fa0c72" containerName="revoke-keys"
Oct 30 03:55:10.914250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.914223    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:10.934492 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:10.930836    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b267f8-openshift-default-dockercfg-r8jvd\""
Oct 30 03:55:11.025371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.025331    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"]
Oct 30 03:55:11.036915 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.036880    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.039675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.039639    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.039916 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.039895    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.040140 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.040119    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.040398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.040374    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjx4z\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.040576 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.040557    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.040756 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.040739    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.040889 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.040869    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.041093 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.041073    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.409126    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.409315    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.409509    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.409608    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.409741    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjx4z\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.409964    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.410052    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.410088    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.411479 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.410150    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.417689 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.416903    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.458414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.457967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.458414 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.458219    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.458930 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.458772    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.461974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.461274    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.479145 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.479025    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.479329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.479248    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.479329 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.479262    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.485736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.485704    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjx4z\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:11.486593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.486447    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz" event={"ID":"781be59f-c261-49e7-a461-ff1ba0fa0c72","Type":"ContainerDied","Data":"fec759a6d5bdc52c55a0d6f5220eed0972b998fbfd2ee7f2a47479dfab794fd6"}
Oct 30 03:55:11.486593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.486537    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fec759a6d5bdc52c55a0d6f5220eed0972b998fbfd2ee7f2a47479dfab794fd6"
Oct 30 03:55:11.487390 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.486966    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"
Oct 30 03:55:11.717908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:11.717813    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:12.048211 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:55:12.048176    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b8fedb_3066_4c44_8e86_d3f72dee551a.slice/crio-0172f2195743f99ba14b96f37457a620137ec830aa2aa282f6019fd3aa4aaf46 WatchSource:0}: Error finding container 0172f2195743f99ba14b96f37457a620137ec830aa2aa282f6019fd3aa4aaf46: Status 404 returned error can't find the container with id 0172f2195743f99ba14b96f37457a620137ec830aa2aa282f6019fd3aa4aaf46
Oct 30 03:55:12.049297 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.049268    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"]
Oct 30 03:55:12.051462 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.051395    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:55:12.051538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.051477    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:55:12.051538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.051509    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:55:12.501621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.501521    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" event={"ID":"c7b8fedb-3066-4c44-8e86-d3f72dee551a","Type":"ContainerStarted","Data":"f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"}
Oct 30 03:55:12.501621 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.501592    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" event={"ID":"c7b8fedb-3066-4c44-8e86-d3f72dee551a","Type":"ContainerStarted","Data":"0172f2195743f99ba14b96f37457a620137ec830aa2aa282f6019fd3aa4aaf46"}
Oct 30 03:55:12.555441 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.550849    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podStartSLOduration=2.550832732 podStartE2EDuration="2.550832732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:12.547831099 +0000 UTC m=+3140.976720543" watchObservedRunningTime="2026-10-30 03:55:12.550832732 +0000 UTC m=+3140.979722151"
Oct 30 03:55:12.721447 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:12.720520    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:13.726963 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:13.726934    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:14.646739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:14.638237    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:14.646739 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:14.645678    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:55:15.667630 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:15.667587    4877 patch_prober.go:29] interesting pod/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:55:15.668012 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:15.667653    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:55:16.231465 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:55:16.231407    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:55:17.666656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:17.666627    4877 generic.go:356] "Generic (PLEG): container finished" podID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerID="8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee" exitCode=0
Oct 30 03:55:17.667050 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:17.666707    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" event={"ID":"636f9bc8-0c75-4a86-8a8e-67d950e052d0","Type":"ContainerDied","Data":"8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"}
Oct 30 03:55:17.667322 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:17.667299    4877 scope.go:122] "RemoveContainer" containerID="8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"
Oct 30 03:55:17.669610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:17.669581    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:55:17.669729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:17.669669    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:55:17.669729 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:17.669716    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:55:18.033037 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:18.032998    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"]
Oct 30 03:55:18.038857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:18.038828    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37532b3a-4n7f4"]
Oct 30 03:55:19.141168 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.141132    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"]
Oct 30 03:55:19.157151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.157123    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"]
Oct 30 03:55:19.157308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.157207    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.157807 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.157782    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" event={"ID":"636f9bc8-0c75-4a86-8a8e-67d950e052d0","Type":"ContainerStarted","Data":"cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d"}
Oct 30 03:55:19.170207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.170185    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b267f8\""
Oct 30 03:55:19.206568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.201368    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b267f8-dockercfg-ggwpk\""
Oct 30 03:55:19.237985 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.237942    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72mn\" (UniqueName: \"kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn\") pod \"payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.238208 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.238098    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.340351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.340133    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72mn\" (UniqueName: \"kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn\") pod \"payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.340351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.340241    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.341010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.340989    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.358009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.357982    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72mn\" (UniqueName: \"kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn\") pod \"payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.471480 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.471386    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:19.732537 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.731915    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ffe568-b300-4f22-ad1b-41fdf86d0fab" path="/var/lib/kubelet/pods/b1ffe568-b300-4f22-ad1b-41fdf86d0fab/volumes"
Oct 30 03:55:19.873621 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:55:19.873591    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc76012_2888_4c34_bbe8_5a3b97fb2ff0.slice/crio-5f8f4287abac794646c67adf6de4e20e2c42ce7e07a9a6979d20799b3ea707db WatchSource:0}: Error finding container 5f8f4287abac794646c67adf6de4e20e2c42ce7e07a9a6979d20799b3ea707db: Status 404 returned error can't find the container with id 5f8f4287abac794646c67adf6de4e20e2c42ce7e07a9a6979d20799b3ea707db
Oct 30 03:55:19.874884 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.874858    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"]
Oct 30 03:55:19.915673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.915281    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:55:19.916328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.916214    4877 patch_prober.go:29] interesting pod/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 03:55:19.916328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:19.916274    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 03:55:20.170009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:20.169974    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" event={"ID":"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0","Type":"ContainerStarted","Data":"830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34"}
Oct 30 03:55:20.170438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:20.170037    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" event={"ID":"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0","Type":"ContainerStarted","Data":"5f8f4287abac794646c67adf6de4e20e2c42ce7e07a9a6979d20799b3ea707db"}
Oct 30 03:55:20.170438 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:20.170067    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:20.911019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:20.910980    4877 patch_prober.go:29] interesting pod/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 03:55:20.911250 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:20.911044    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 03:55:21.923486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:21.923023    4877 patch_prober.go:29] interesting pod/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 03:55:21.923486 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:21.923106    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 03:55:22.910456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:22.910401    4877 patch_prober.go:29] interesting pod/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 03:55:22.910797 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:22.910626    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 03:55:23.911134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:23.911097    4877 patch_prober.go:29] interesting pod/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Oct 30 03:55:23.911551 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:23.911167    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Oct 30 03:55:24.941688 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:24.941648    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:55:24.943852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:24.943826    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:55:24.945100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:24.945081    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 03:55:24.993750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:24.993684    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" podStartSLOduration=6.993667878 podStartE2EDuration="6.993667878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:20.257320898 +0000 UTC m=+3148.686210341" watchObservedRunningTime="2026-10-30 03:55:24.993667878 +0000 UTC m=+3153.422557296"
Oct 30 03:55:29.541562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.541524    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"]
Oct 30 03:55:29.546721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.546702    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.564034 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.564006    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"]
Oct 30 03:55:29.566970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.566946    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:55:29.567072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.567036    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:55:29.683794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.683757    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.683999 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.683829    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74jk\" (UniqueName: \"kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.684098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.684079    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.786947 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.786892    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74jk\" (UniqueName: \"kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.787357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.787332    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.787606 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.787562    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.789221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.789193    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.791610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.791547    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.807046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.807015    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74jk\" (UniqueName: \"kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk\") pod \"maas-api-revoke-sub-1318ae60-v79dc\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:29.862538 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:29.862508    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:30.169277 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:55:30.169243    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990e7fc4_0c89_45f9_88fb_a118b6df9796.slice/crio-0c56dcfef77288420034a3e3debbd8447f149ebe50b50cbfc238374dcb54a6c3 WatchSource:0}: Error finding container 0c56dcfef77288420034a3e3debbd8447f149ebe50b50cbfc238374dcb54a6c3: Status 404 returned error can't find the container with id 0c56dcfef77288420034a3e3debbd8447f149ebe50b50cbfc238374dcb54a6c3
Oct 30 03:55:30.171211 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:30.171186    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"]
Oct 30 03:55:30.507701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:30.507618    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc" event={"ID":"990e7fc4-0c89-45f9-88fb-a118b6df9796","Type":"ContainerStarted","Data":"0c56dcfef77288420034a3e3debbd8447f149ebe50b50cbfc238374dcb54a6c3"}
Oct 30 03:55:31.245448 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:31.244575    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 03:55:31.526011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:31.525970    4877 generic.go:356] "Generic (PLEG): container finished" podID="990e7fc4-0c89-45f9-88fb-a118b6df9796" containerID="8256089c97342d343e6b8fd0944ca64c87f4c9f4ab44523df8b98226aeb9978e" exitCode=0
Oct 30 03:55:31.526323 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:31.526303    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc" event={"ID":"990e7fc4-0c89-45f9-88fb-a118b6df9796","Type":"ContainerDied","Data":"8256089c97342d343e6b8fd0944ca64c87f4c9f4ab44523df8b98226aeb9978e"}
Oct 30 03:55:32.785148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.785118    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:32.912794 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.912695    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca\") pod \"990e7fc4-0c89-45f9-88fb-a118b6df9796\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") "
Oct 30 03:55:32.912986 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.912819    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk\" (UniqueName: \"kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk\") pod \"990e7fc4-0c89-45f9-88fb-a118b6df9796\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") "
Oct 30 03:55:32.913097 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.913075    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca\") pod \"990e7fc4-0c89-45f9-88fb-a118b6df9796\" (UID: \"990e7fc4-0c89-45f9-88fb-a118b6df9796\") "
Oct 30 03:55:32.913162 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.913126    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca" pod "990e7fc4-0c89-45f9-88fb-a118b6df9796" (UID: "990e7fc4-0c89-45f9-88fb-a118b6df9796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:32.914067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.914036    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:32.915331 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.915287    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk" pod "990e7fc4-0c89-45f9-88fb-a118b6df9796" (UID: "990e7fc4-0c89-45f9-88fb-a118b6df9796"). InnerVolumeSpecName "kube-api-access-f74jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:32.915590 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:32.915566    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca" pod "990e7fc4-0c89-45f9-88fb-a118b6df9796" (UID: "990e7fc4-0c89-45f9-88fb-a118b6df9796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:55:33.034476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:33.031907    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74jk\" (UniqueName: \"kubernetes.io/projected/990e7fc4-0c89-45f9-88fb-a118b6df9796-kube-api-access-f74jk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:33.034476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:33.031951    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/990e7fc4-0c89-45f9-88fb-a118b6df9796-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:33.581242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:33.581199    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc" event={"ID":"990e7fc4-0c89-45f9-88fb-a118b6df9796","Type":"ContainerDied","Data":"0c56dcfef77288420034a3e3debbd8447f149ebe50b50cbfc238374dcb54a6c3"}
Oct 30 03:55:33.581649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:33.581629    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c56dcfef77288420034a3e3debbd8447f149ebe50b50cbfc238374dcb54a6c3"
Oct 30 03:55:33.581793 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:33.581538    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"
Oct 30 03:55:44.665653 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:44.665615    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"]
Oct 30 03:55:44.668279 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:44.666355    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy" containerID="cri-o://71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0" gracePeriod=30
Oct 30 03:55:46.257122 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:55:46.257081    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:55:47.113746 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.113708    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"]
Oct 30 03:55:47.114757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.114731    4877 cpu_manager.go:414] "containerMap: removing container" podUID="990e7fc4-0c89-45f9-88fb-a118b6df9796" containerName="revoke-keys"
Oct 30 03:55:47.114757 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.114753    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990e7fc4-0c89-45f9-88fb-a118b6df9796" containerName="revoke-keys"
Oct 30 03:55:47.115039 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.115024    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="990e7fc4-0c89-45f9-88fb-a118b6df9796" containerName="revoke-keys"
Oct 30 03:55:47.119417 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.119398    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.128760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.128724    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"]
Oct 30 03:55:47.131629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.131605    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:55:47.131764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.131745    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:55:47.231668 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.231633    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzpk8\" (UniqueName: \"kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.231859 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.231687    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.232041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.232015    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.334006 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.333960    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.334566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.334065    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzpk8\" (UniqueName: \"kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.334566 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.334118    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.334814 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.334792    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.336495 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.336471    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.352443 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.352399    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzpk8\" (UniqueName: \"kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8\") pod \"maas-api-revoke-sub-11c379d9-hqjrw\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.440357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.440280    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:47.848289 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.848260    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"]
Oct 30 03:55:47.996927 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:47.996888    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw" event={"ID":"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9","Type":"ContainerStarted","Data":"59a1d76b8add5b55b17f03bceefafc6bbf4e9257969017abf732691635eee7e4"}
Oct 30 03:55:49.009129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:49.009095    4877 generic.go:356] "Generic (PLEG): container finished" podID="d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" containerID="b1b44e19c37e7188eeef3e144a91e385ea6d4097a048780437b4718edc4371b7" exitCode=0
Oct 30 03:55:49.009542 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:49.009143    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw" event={"ID":"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9","Type":"ContainerDied","Data":"b1b44e19c37e7188eeef3e144a91e385ea6d4097a048780437b4718edc4371b7"}
Oct 30 03:55:50.001618 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.001592    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:55:50.020184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.020148    4877 generic.go:356] "Generic (PLEG): container finished" podID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerID="71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0" exitCode=0
Oct 30 03:55:50.020558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.020191    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" event={"ID":"0f1620d8-851c-4b2e-bacf-be013a1cb957","Type":"ContainerDied","Data":"71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0"}
Oct 30 03:55:50.020558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.020220    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"
Oct 30 03:55:50.020558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.020248    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm" event={"ID":"0f1620d8-851c-4b2e-bacf-be013a1cb957","Type":"ContainerDied","Data":"9c1c0b460a563fb6a036b9adeb859f9c1f164f19faa59538dc421b9eddba17cb"}
Oct 30 03:55:50.020558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.020270    4877 scope.go:122] "RemoveContainer" containerID="71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0"
Oct 30 03:55:50.036108 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.036087    4877 scope.go:122] "RemoveContainer" containerID="b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"
Oct 30 03:55:50.053605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.053580    4877 scope.go:122] "RemoveContainer" containerID="71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0"
Oct 30 03:55:50.054045 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:55:50.054018    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0\": container with ID starting with 71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0 not found: ID does not exist" containerID="71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0"
Oct 30 03:55:50.054148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.054058    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0"} err="failed to get container status \"71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0\": rpc error: code = NotFound desc = could not find container \"71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0\": container with ID starting with 71878beab05838373afc729f4903c982379958c7803fe64834ce112de6dcfbc0 not found: ID does not exist"
Oct 30 03:55:50.054148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.054083    4877 scope.go:122] "RemoveContainer" containerID="b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"
Oct 30 03:55:50.054393 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:55:50.054370    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710\": container with ID starting with b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710 not found: ID does not exist" containerID="b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"
Oct 30 03:55:50.054489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.054403    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710"} err="failed to get container status \"b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710\": rpc error: code = NotFound desc = could not find container \"b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710\": container with ID starting with b9c3b3f4beeb02c660679310d4ae2a43b8e373451f163cf7bb888188b459c710 not found: ID does not exist"
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107418    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107568    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107612    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107705    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107772    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107831    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107886    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107978    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.107990    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.108068    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.108103    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket\") pod \"0f1620d8-851c-4b2e-bacf-be013a1cb957\" (UID: \"0f1620d8-851c-4b2e-bacf-be013a1cb957\") "
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.108848    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109081    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109126    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109278    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109295    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109308    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109317    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1620d8-851c-4b2e-bacf-be013a1cb957-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.110291 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.109325    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.113031 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.113002    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:55:50.113163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.113038    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:50.113163 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.113075    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "kube-api-access-24zlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:50.114079 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.114055    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy" pod "0f1620d8-851c-4b2e-bacf-be013a1cb957" (UID: "0f1620d8-851c-4b2e-bacf-be013a1cb957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:50.211562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.211526    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zlz\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-kube-api-access-24zlz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.211562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.211553    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.211562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.211565    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.211810 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.211578    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1620d8-851c-4b2e-bacf-be013a1cb957-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.224483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.224451    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:50.312977 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.312933    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8\" (UniqueName: \"kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8\") pod \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") "
Oct 30 03:55:50.313671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.313097    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca\") pod \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") "
Oct 30 03:55:50.313671 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.313270    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca\") pod \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\" (UID: \"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9\") "
Oct 30 03:55:50.313992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.313945    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca" pod "d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" (UID: "d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:50.314375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.314328    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.315909 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.315884    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca" pod "d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" (UID: "d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:55:50.316483 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.316417    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8" pod "d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" (UID: "d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9"). InnerVolumeSpecName "kube-api-access-gzpk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:50.374655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.374621    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"]
Oct 30 03:55:50.380908 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.380883    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ae0e75-openshift-default-545f855fd9-7n5bm"]
Oct 30 03:55:50.416509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.416471    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzpk8\" (UniqueName: \"kubernetes.io/projected/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-kube-api-access-gzpk8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:50.416509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:50.416497    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:55:51.033028 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:51.033000    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"
Oct 30 03:55:51.033541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:51.033050    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw" event={"ID":"d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9","Type":"ContainerDied","Data":"59a1d76b8add5b55b17f03bceefafc6bbf4e9257969017abf732691635eee7e4"}
Oct 30 03:55:51.033541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:51.033112    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59a1d76b8add5b55b17f03bceefafc6bbf4e9257969017abf732691635eee7e4"
Oct 30 03:55:51.547974 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:51.547941    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"]
Oct 30 03:55:51.625100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:51.625068    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"]
Oct 30 03:55:51.703848 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:51.703817    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" path="/var/lib/kubelet/pods/0f1620d8-851c-4b2e-bacf-be013a1cb957/volumes"
Oct 30 03:55:54.124245 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:54.124204    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"]
Oct 30 03:55:54.134351 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:54.134318    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48d4ea2c-c4dbv"]
Oct 30 03:55:55.701246 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:55.701210    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd" path="/var/lib/kubelet/pods/d0bdb2d1-b9e3-49fb-9629-5a8f835f75dd/volumes"
Oct 30 03:55:56.011804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:56.011724    4877 scope.go:122] "RemoveContainer" containerID="8630c9aad7f264e6929a9c7f795aae13b02ecd00aab4ccbb330ca42f44b71141"
Oct 30 03:55:56.027375 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:56.027350    4877 scope.go:122] "RemoveContainer" containerID="09ae3ab46c64205aa961c61ad576abf5288331b2994029d794f4d44252f4c0d8"
Oct 30 03:55:56.548383 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:56.548336    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing" containerID="cri-o://0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab" gracePeriod=25
Oct 30 03:55:56.625396 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:56.625349    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing" containerID="cri-o://f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759" gracePeriod=25
Oct 30 03:55:56.851567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:56.851481    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:55:56.854212 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:55:56.854179    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:56:06.850726 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:06.850632    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:56:06.851132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:06.850767    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:56:07.029787 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:07.029755    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"]
Oct 30 03:56:07.036926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:07.036894    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b29971b-bzb7t"]
Oct 30 03:56:07.705285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:07.705249    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31b8385-d63e-47e3-8b43-72bd558b479b" path="/var/lib/kubelet/pods/e31b8385-d63e-47e3-8b43-72bd558b479b/volumes"
Oct 30 03:56:16.084731 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:16.076120    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 03:56:16.302657 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:16.302595    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:56:16.872249 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:56:16.854013    4877 logging.go:55] [core] [Channel #11869 SubChannel #11870]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.0:50052", ServerName: "10.133.1.0:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.0:50052: connect: connection refused"
Oct 30 03:56:17.046626 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.046380    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"]
Oct 30 03:56:17.055063 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.055030    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edf68f22-vwr4z"]
Oct 30 03:56:17.056673 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.056322    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:56:17.160567 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.160536    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume\") pod \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") "
Oct 30 03:56:17.161065 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.160877    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk\" (UniqueName: \"kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk\") pod \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\" (UID: \"9b99d5ad-c664-4fae-b8f8-ea76535a03c5\") "
Oct 30 03:56:17.161173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.161142    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume" pod "9b99d5ad-c664-4fae-b8f8-ea76535a03c5" (UID: "9b99d5ad-c664-4fae-b8f8-ea76535a03c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:17.161685 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.161659    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:17.164198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.164172    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk" pod "9b99d5ad-c664-4fae-b8f8-ea76535a03c5" (UID: "9b99d5ad-c664-4fae-b8f8-ea76535a03c5"). InnerVolumeSpecName "kube-api-access-xs6tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:17.164896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.164873    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:56:17.263493 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.263450    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume\") pod \"74b372d9-e957-4c98-ad09-a114e9c0e387\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") "
Oct 30 03:56:17.263708 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.263548    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x\" (UniqueName: \"kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x\") pod \"74b372d9-e957-4c98-ad09-a114e9c0e387\" (UID: \"74b372d9-e957-4c98-ad09-a114e9c0e387\") "
Oct 30 03:56:17.264249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.264219    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume" pod "74b372d9-e957-4c98-ad09-a114e9c0e387" (UID: "74b372d9-e957-4c98-ad09-a114e9c0e387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:17.264469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.264406    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6tk\" (UniqueName: \"kubernetes.io/projected/9b99d5ad-c664-4fae-b8f8-ea76535a03c5-kube-api-access-xs6tk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:17.266836 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.266801    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x" pod "74b372d9-e957-4c98-ad09-a114e9c0e387" (UID: "74b372d9-e957-4c98-ad09-a114e9c0e387"). InnerVolumeSpecName "kube-api-access-46r2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:17.366153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.366116    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b372d9-e957-4c98-ad09-a114e9c0e387-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:17.366153 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.366155    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46r2x\" (UniqueName: \"kubernetes.io/projected/74b372d9-e957-4c98-ad09-a114e9c0e387-kube-api-access-46r2x\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:17.620959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.620917    4877 generic.go:356] "Generic (PLEG): container finished" podID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerID="f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759" exitCode=0
Oct 30 03:56:17.621129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.620978    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"
Oct 30 03:56:17.621129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.621004    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" event={"ID":"9b99d5ad-c664-4fae-b8f8-ea76535a03c5","Type":"ContainerDied","Data":"f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759"}
Oct 30 03:56:17.621129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.621063    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" event={"ID":"9b99d5ad-c664-4fae-b8f8-ea76535a03c5","Type":"ContainerDied","Data":"08197d81e27ba79925489091da700794cbfd1e3f7daa5dd33bac054235c75e80"}
Oct 30 03:56:17.621129 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.621089    4877 scope.go:122] "RemoveContainer" containerID="f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759"
Oct 30 03:56:17.622847 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.622819    4877 generic.go:356] "Generic (PLEG): container finished" podID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerID="0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab" exitCode=0
Oct 30 03:56:17.623010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.622853    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" event={"ID":"74b372d9-e957-4c98-ad09-a114e9c0e387","Type":"ContainerDied","Data":"0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab"}
Oct 30 03:56:17.623010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.622909    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" event={"ID":"74b372d9-e957-4c98-ad09-a114e9c0e387","Type":"ContainerDied","Data":"81f7fac442e71c696f38b8891feead811e643333c95c30652980b6f303672ae9"}
Oct 30 03:56:17.623010 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.622862    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"
Oct 30 03:56:17.710464 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.706198    4877 scope.go:122] "RemoveContainer" containerID="f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759"
Oct 30 03:56:17.714110 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:17.713783    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759\": container with ID starting with f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759 not found: ID does not exist" containerID="f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759"
Oct 30 03:56:17.714110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.713832    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759"} err="failed to get container status \"f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759\": rpc error: code = NotFound desc = could not find container \"f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759\": container with ID starting with f07a96621f225b3a247ecfe0211bb349424528cdf184a6cacf1a2677f375d759 not found: ID does not exist"
Oct 30 03:56:17.714110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.713860    4877 scope.go:122] "RemoveContainer" containerID="0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab"
Oct 30 03:56:17.739268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.739235    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5771a794-0c7b-4d0b-b327-368d8af9980e" path="/var/lib/kubelet/pods/5771a794-0c7b-4d0b-b327-368d8af9980e/volumes"
Oct 30 03:56:17.739885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.739858    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"]
Oct 30 03:56:17.746000 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.745971    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h"]
Oct 30 03:56:17.748548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.748339    4877 scope.go:122] "RemoveContainer" containerID="0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab"
Oct 30 03:56:17.748779 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:17.748756    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab\": container with ID starting with 0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab not found: ID does not exist" containerID="0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab"
Oct 30 03:56:17.748917 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.748788    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab"} err="failed to get container status \"0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab\": rpc error: code = NotFound desc = could not find container \"0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab\": container with ID starting with 0117b8422f849e6c3b4efec64bf9bd3a0892c59038cc683ec70ceb9b15a028ab not found: ID does not exist"
Oct 30 03:56:17.763629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.763600    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"]
Oct 30 03:56:17.769176 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.769152    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42"]
Oct 30 03:56:17.851966 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.851873    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e84013-7b65d6cb4c-fgz8h" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.255:50052\" within 1s: context deadline exceeded"
Oct 30 03:56:17.852218 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:56:17.851956    4877 logging.go:55] [core] [Channel #11867 SubChannel #11868]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.255:50052", ServerName: "10.133.0.255:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.255:50052: operation was canceled"
Oct 30 03:56:17.854418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:17.854383    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e84013-75b759bfb4-dtf42" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.0:50052\" within 1s: context deadline exceeded"
Oct 30 03:56:17.855538 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:56:17.855517    4877 logging.go:55] [core] [Channel #11869 SubChannel #11870]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.0:50052", ServerName: "10.133.1.0:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.0:50052: operation was canceled"
Oct 30 03:56:19.702881 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:19.702843    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" path="/var/lib/kubelet/pods/74b372d9-e957-4c98-ad09-a114e9c0e387/volumes"
Oct 30 03:56:19.703367 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:19.703341    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" path="/var/lib/kubelet/pods/9b99d5ad-c664-4fae-b8f8-ea76535a03c5/volumes"
Oct 30 03:56:28.028973 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:28.028926    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"]
Oct 30 03:56:28.038308 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:28.038282    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1bafeb-6bd6k"]
Oct 30 03:56:29.701178 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:29.701144    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2" path="/var/lib/kubelet/pods/16d5571e-a3b5-46fe-ad7a-2dc47cf50bf2/volumes"
Oct 30 03:56:42.589457 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:42.589394    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"]
Oct 30 03:56:42.589959 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:42.589765    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" containerID="cri-o://f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12" gracePeriod=30
Oct 30 03:56:42.596604 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:42.596579    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"]
Oct 30 03:56:42.596943 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:42.596915    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy" containerID="cri-o://3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557" gracePeriod=30
Oct 30 03:56:43.574641 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:43.574597    4877 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:56:43.574795 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:43.574655    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:56:44.637875 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:44.637834    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"]
Oct 30 03:56:44.638395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:44.638115    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy" containerID="cri-o://b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82" gracePeriod=30
Oct 30 03:56:46.231502 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:46.231466    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:56:47.938344 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:47.937968    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:56:47.967849 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:47.967825    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:56:48.062751 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.062250    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.063230 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.062835    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.063498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.062711    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.063498 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.063193    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.064356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064324    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.064585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064390    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.064585 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064457    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.064809 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064784    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.064850 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064533    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.064895 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064864    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.064979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.064963    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.065046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065000    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.065113 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065052    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.065173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065126    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.065173 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065161    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.065293 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065196    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.065355 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065289    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.065418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065354    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.065418 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065445    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.065593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065507    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.065593 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065576    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq\") pod \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\" (UID: \"e13a6e55-0c25-48e7-a08c-765c3c57ef77\") "
Oct 30 03:56:48.065717 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.065701    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert\") pod \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\" (UID: \"e46386d6-80fd-40a9-aff1-1db047cfb8b9\") "
Oct 30 03:56:48.066191 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.066166    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.066408 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.066386    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.066629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.066525    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:48.066707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.066688    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:48.066829 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.066808    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.066897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.066848    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073603    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073617    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073666    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "kube-api-access-fx6hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.067341    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073778    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073787    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073821    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "kube-api-access-tqsz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073823    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073856    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073859    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073871    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073885    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073903    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073918    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073932    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.074131 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.073945    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.091768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.091742    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy" pod "e46386d6-80fd-40a9-aff1-1db047cfb8b9" (UID: "e46386d6-80fd-40a9-aff1-1db047cfb8b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:48.094788 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.094756    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo" pod "e13a6e55-0c25-48e7-a08c-765c3c57ef77" (UID: "e13a6e55-0c25-48e7-a08c-765c3c57ef77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:56:48.154964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.154882    4877 generic.go:356] "Generic (PLEG): container finished" podID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerID="f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12" exitCode=0
Oct 30 03:56:48.154964 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.154939    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"
Oct 30 03:56:48.155151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.154970    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" event={"ID":"e46386d6-80fd-40a9-aff1-1db047cfb8b9","Type":"ContainerDied","Data":"f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12"}
Oct 30 03:56:48.155151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.155062    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc" event={"ID":"e46386d6-80fd-40a9-aff1-1db047cfb8b9","Type":"ContainerDied","Data":"1971a272ebd5294a90b0443257e921090308407f901ca55d120130f09fcb46df"}
Oct 30 03:56:48.155151 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.155102    4877 scope.go:122] "RemoveContainer" containerID="f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12"
Oct 30 03:56:48.156588 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.156562    4877 generic.go:356] "Generic (PLEG): container finished" podID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerID="3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557" exitCode=0
Oct 30 03:56:48.156727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.156617    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"
Oct 30 03:56:48.156727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.156637    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" event={"ID":"e13a6e55-0c25-48e7-a08c-765c3c57ef77","Type":"ContainerDied","Data":"3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557"}
Oct 30 03:56:48.156727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.156680    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j" event={"ID":"e13a6e55-0c25-48e7-a08c-765c3c57ef77","Type":"ContainerDied","Data":"dc929491a6462896e836c492a2a7880507911e8e1f5e55dbcfdda1d2deffde1d"}
Oct 30 03:56:48.169314 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.169296    4877 scope.go:122] "RemoveContainer" containerID="06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"
Oct 30 03:56:48.176263 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176239    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6hq\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-kube-api-access-fx6hq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176269    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176284    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176296    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176309    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e46386d6-80fd-40a9-aff1-1db047cfb8b9-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176321    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqsz7\" (UniqueName: \"kubernetes.io/projected/e46386d6-80fd-40a9-aff1-1db047cfb8b9-kube-api-access-tqsz7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176333    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.176349 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.176346    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13a6e55-0c25-48e7-a08c-765c3c57ef77-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:48.183328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.183311    4877 scope.go:122] "RemoveContainer" containerID="f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12"
Oct 30 03:56:48.183689 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:48.183659    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12\": container with ID starting with f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12 not found: ID does not exist" containerID="f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12"
Oct 30 03:56:48.183799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.183698    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12"} err="failed to get container status \"f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12\": rpc error: code = NotFound desc = could not find container \"f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12\": container with ID starting with f649ff4416d634b6022e358cb29f604b57ce8c8a99679e49aeb8373d97015a12 not found: ID does not exist"
Oct 30 03:56:48.183799 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.183722    4877 scope.go:122] "RemoveContainer" containerID="06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"
Oct 30 03:56:48.184052 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:48.184026    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1\": container with ID starting with 06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1 not found: ID does not exist" containerID="06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"
Oct 30 03:56:48.184109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.184062    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1"} err="failed to get container status \"06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1\": rpc error: code = NotFound desc = could not find container \"06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1\": container with ID starting with 06fc6823acb147b988d566958a4d1e3a98114a4cb003b03bf0b9ee3ea7ace7b1 not found: ID does not exist"
Oct 30 03:56:48.184109 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.184084    4877 scope.go:122] "RemoveContainer" containerID="3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557"
Oct 30 03:56:48.197816 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.197784    4877 scope.go:122] "RemoveContainer" containerID="3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557"
Oct 30 03:56:48.198093 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:48.198070    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557\": container with ID starting with 3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557 not found: ID does not exist" containerID="3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557"
Oct 30 03:56:48.198150 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.198104    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557"} err="failed to get container status \"3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557\": rpc error: code = NotFound desc = could not find container \"3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557\": container with ID starting with 3c3c4ddd2935a9f89760d387fd6fe7dfc1f7b687fdcde4f6fdac3a6975be9557 not found: ID does not exist"
Oct 30 03:56:48.219027 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.218998    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"]
Oct 30 03:56:48.227343 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.227318    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-ldw7j"]
Oct 30 03:56:48.246857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.246836    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"]
Oct 30 03:56:48.251451 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:48.251416    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-z2jqc"]
Oct 30 03:56:49.703657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:49.703626    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" path="/var/lib/kubelet/pods/e13a6e55-0c25-48e7-a08c-765c3c57ef77/volumes"
Oct 30 03:56:49.704200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:49.704186    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" path="/var/lib/kubelet/pods/e46386d6-80fd-40a9-aff1-1db047cfb8b9/volumes"
Oct 30 03:56:49.965044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:49.965016    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:56:50.012527 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.012491    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.012705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.012551    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.012705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.012603    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.012705 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.012683    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.012891 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.012854    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.012991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.012972    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.013072 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.013051    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.013138 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.013084    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.013202 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.013155    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token\") pod \"f68516de-5dcd-4571-8208-937bddb2108c\" (UID: \"f68516de-5dcd-4571-8208-937bddb2108c\") "
Oct 30 03:56:50.015743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.015241    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:50.015743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.015547    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:50.015743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.015714    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:50.016501 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.016476    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:50.016711 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.016675    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:50.017476 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.017453    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.017609 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.017596    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.017701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.017690    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.017792 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.017782    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.017876 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.017866    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f68516de-5dcd-4571-8208-937bddb2108c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.026707 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.026673    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:50.026822 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.026799    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:50.027402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.027351    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "kube-api-access-hrgkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:50.027402 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.027361    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo" pod "f68516de-5dcd-4571-8208-937bddb2108c" (UID: "f68516de-5dcd-4571-8208-937bddb2108c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:56:50.119627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.119591    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f68516de-5dcd-4571-8208-937bddb2108c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.119627 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.119626    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f68516de-5dcd-4571-8208-937bddb2108c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.119852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.119642    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.119852 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.119657    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrgkg\" (UniqueName: \"kubernetes.io/projected/f68516de-5dcd-4571-8208-937bddb2108c-kube-api-access-hrgkg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:56:50.181678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.181592    4877 generic.go:356] "Generic (PLEG): container finished" podID="f68516de-5dcd-4571-8208-937bddb2108c" containerID="b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82" exitCode=0
Oct 30 03:56:50.181678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.181641    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" event={"ID":"f68516de-5dcd-4571-8208-937bddb2108c","Type":"ContainerDied","Data":"b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82"}
Oct 30 03:56:50.181678 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.181653    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"
Oct 30 03:56:50.181972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.181692    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48" event={"ID":"f68516de-5dcd-4571-8208-937bddb2108c","Type":"ContainerDied","Data":"7e340e741621901de6f5fd28a468d9c2a55f81de92ce3c34645adab96f04c01f"}
Oct 30 03:56:50.181972 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.181714    4877 scope.go:122] "RemoveContainer" containerID="b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82"
Oct 30 03:56:50.196156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.196136    4877 scope.go:122] "RemoveContainer" containerID="14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"
Oct 30 03:56:50.210314 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.210289    4877 scope.go:122] "RemoveContainer" containerID="b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82"
Oct 30 03:56:50.210678 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:50.210650    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82\": container with ID starting with b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82 not found: ID does not exist" containerID="b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82"
Oct 30 03:56:50.210766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.210689    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82"} err="failed to get container status \"b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82\": rpc error: code = NotFound desc = could not find container \"b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82\": container with ID starting with b3534d9e28827ed25aed506916c6ab5495516a52b82b416905448299eae8ae82 not found: ID does not exist"
Oct 30 03:56:50.210766 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.210711    4877 scope.go:122] "RemoveContainer" containerID="14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"
Oct 30 03:56:50.211044 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:56:50.211023    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e\": container with ID starting with 14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e not found: ID does not exist" containerID="14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"
Oct 30 03:56:50.211088 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.211051    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e"} err="failed to get container status \"14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e\": rpc error: code = NotFound desc = could not find container \"14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e\": container with ID starting with 14911f6a28d360c0f230200a64fd6ea407d504cd9be2cc74b294686259c3ef7e not found: ID does not exist"
Oct 30 03:56:50.233023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.232996    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"]
Oct 30 03:56:50.239815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:50.239792    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-5nl48"]
Oct 30 03:56:51.705122 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:51.705088    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68516de-5dcd-4571-8208-937bddb2108c" path="/var/lib/kubelet/pods/f68516de-5dcd-4571-8208-937bddb2108c/volumes"
Oct 30 03:56:56.073100 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:56.073065    4877 scope.go:122] "RemoveContainer" containerID="18181811fcf4cd719fcac3f2680ede22791abef409244a5114e45cbb946ca673"
Oct 30 03:56:56.114257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:56:56.114222    4877 scope.go:122] "RemoveContainer" containerID="48192a98df0ad83c51f19cd776d52486e1d0a069b1efca914cec5ebba90b8694"
Oct 30 03:57:00.680629 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:00.680573    4877 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 03:57:00.681110 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:00.680674    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:57:04.423121 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.423023    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"]
Oct 30 03:57:04.423993 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.423960    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy" containerID="cri-o://b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586" gracePeriod=30
Oct 30 03:57:04.439512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.431370    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"]
Oct 30 03:57:04.439512 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.431630    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy" containerID="cri-o://e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775" gracePeriod=30
Oct 30 03:57:04.835227 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.835143    4877 generic.go:356] "Generic (PLEG): container finished" podID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerID="f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9" exitCode=0
Oct 30 03:57:04.835801 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.835709    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" event={"ID":"c7b8fedb-3066-4c44-8e86-d3f72dee551a","Type":"ContainerDied","Data":"f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"}
Oct 30 03:57:04.837584 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.837557    4877 scope.go:122] "RemoveContainer" containerID="f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"
Oct 30 03:57:04.842030 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.841991    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:57:04.842243 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.842212    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:57:04.842354 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:04.842272    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 03:57:05.847541 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:05.847489    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" event={"ID":"c7b8fedb-3066-4c44-8e86-d3f72dee551a","Type":"ContainerStarted","Data":"ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d"}
Oct 30 03:57:06.718760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:06.718725    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:57:06.719011 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:06.718979    4877 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:57:06.719165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:06.719032    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:57:07.718281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:07.718240    4877 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:57:07.718802 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:07.718312    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:57:08.718107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:08.718072    4877 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:57:08.718397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:08.718366    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:57:09.724222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:09.720606    4877 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:57:09.724222 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:09.720698    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:57:09.920198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:09.920145    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:57:09.944649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:09.944616    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:57:10.041545 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.041492    4877 generic.go:356] "Generic (PLEG): container finished" podID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerID="b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586" exitCode=0
Oct 30 03:57:10.060789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.060762    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"
Oct 30 03:57:10.061300 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.061275    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" event={"ID":"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b","Type":"ContainerDied","Data":"b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586"}
Oct 30 03:57:10.061522 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.061496    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5" event={"ID":"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b","Type":"ContainerDied","Data":"40255224e3d0eada88b2cd983d8fc7ce1e3a47b6120204016eacc854d724ec59"}
Oct 30 03:57:10.061679 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.061664    4877 scope.go:122] "RemoveContainer" containerID="b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586"
Oct 30 03:57:10.062505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.062484    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.062670 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.062653    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.062804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.062790    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.062910 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.062897    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.063007 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.062995    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.063148 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.063133    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.063281 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.063266    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.063485 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.063463    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.068826 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.068804    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.069008 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.068993    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.069137 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.069123    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.069286 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.069273    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.069465 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.069451    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.069656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.069639    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.069818 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.069805    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo\") pod \"198b1ec6-ac60-49d8-bda5-581bbb36c219\" (UID: \"198b1ec6-ac60-49d8-bda5-581bbb36c219\") "
Oct 30 03:57:10.069991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.069977    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.071328 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.071310    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.074456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.073628    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket\") pod \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\" (UID: \"c63b3dd3-9d77-4f33-a4a7-024bf6053f4b\") "
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.073129    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.074627    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.074766    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.074926    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.075169    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.075491    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.076534    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "kube-api-access-c9bxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.084642    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.126524    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.126726    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.126924    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.134829    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "kube-api-access-7tdvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.134874    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:10.135535 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.135202    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:57:10.136718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.135804    4877 generic.go:356] "Generic (PLEG): container finished" podID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerID="e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775" exitCode=0
Oct 30 03:57:10.136718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.135991    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" event={"ID":"198b1ec6-ac60-49d8-bda5-581bbb36c219","Type":"ContainerDied","Data":"e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775"}
Oct 30 03:57:10.136718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.136043    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6" event={"ID":"198b1ec6-ac60-49d8-bda5-581bbb36c219","Type":"ContainerDied","Data":"287b37a8c006c7b915f6ac7d70d6d5558499f9bfdc676fd07f973af08e534a80"}
Oct 30 03:57:10.136718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.136124    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"
Oct 30 03:57:10.143080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.137172    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:57:10.143080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.138173    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy" pod "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" (UID: "c63b3dd3-9d77-4f33-a4a7-024bf6053f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.143080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.139653    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:10.143080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.139766    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token" pod "198b1ec6-ac60-49d8-bda5-581bbb36c219" (UID: "198b1ec6-ac60-49d8-bda5-581bbb36c219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:10.159929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.159541    4877 scope.go:122] "RemoveContainer" containerID="f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"
Oct 30 03:57:10.171682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.171607    4877 scope.go:122] "RemoveContainer" containerID="b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586"
Oct 30 03:57:10.171957 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:57:10.171921    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586\": container with ID starting with b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586 not found: ID does not exist" containerID="b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586"
Oct 30 03:57:10.172062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.171959    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586"} err="failed to get container status \"b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586\": rpc error: code = NotFound desc = could not find container \"b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586\": container with ID starting with b0de3e282ea43c887363eb6ad0836beb4b865288fc3a1dae456387f504b08586 not found: ID does not exist"
Oct 30 03:57:10.172062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.171983    4877 scope.go:122] "RemoveContainer" containerID="f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"
Oct 30 03:57:10.172277 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:57:10.172237    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99\": container with ID starting with f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99 not found: ID does not exist" containerID="f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"
Oct 30 03:57:10.172357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.172272    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99"} err="failed to get container status \"f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99\": rpc error: code = NotFound desc = could not find container \"f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99\": container with ID starting with f821a4778989c410852bdb13d4ba10e95e31956f7bd68e4b191ef613581c2d99 not found: ID does not exist"
Oct 30 03:57:10.172357 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.172294    4877 scope.go:122] "RemoveContainer" containerID="e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775"
Oct 30 03:57:10.176242 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176213    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176246    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176262    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176274    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176287    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tdvk\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-kube-api-access-7tdvk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176301    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176314    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176329    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176347 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176343    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176355    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176371    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bxm\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-kube-api-access-c9bxm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176385    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176398    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.176831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.176413    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.213133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.186478    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198b1ec6-ac60-49d8-bda5-581bbb36c219-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.213133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.186501    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198b1ec6-ac60-49d8-bda5-581bbb36c219-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.213133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.186516    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.213133 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.186529    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198b1ec6-ac60-49d8-bda5-581bbb36c219-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:57:10.224326 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.224206    4877 scope.go:122] "RemoveContainer" containerID="e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775"
Oct 30 03:57:10.224657 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:57:10.224589    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775\": container with ID starting with e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775 not found: ID does not exist" containerID="e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775"
Oct 30 03:57:10.224657 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.224630    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775"} err="failed to get container status \"e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775\": rpc error: code = NotFound desc = could not find container \"e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775\": container with ID starting with e592ecb213cff005023d2bac4b2901d5d6c85bea30bb0633d0f640fd03b8d775 not found: ID does not exist"
Oct 30 03:57:10.422575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.422512    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"]
Oct 30 03:57:10.431813 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.431718    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-bp5f5"]
Oct 30 03:57:10.516563 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.514636    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"]
Oct 30 03:57:10.520255 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.520223    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-ktnm6"]
Oct 30 03:57:10.719215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.719112    4877 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 30 03:57:10.719215 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:10.719184    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 30 03:57:11.721290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:11.721255    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" path="/var/lib/kubelet/pods/198b1ec6-ac60-49d8-bda5-581bbb36c219/volumes"
Oct 30 03:57:11.721926 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:11.721906    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" path="/var/lib/kubelet/pods/c63b3dd3-9d77-4f33-a4a7-024bf6053f4b/volumes"
Oct 30 03:57:11.724561 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:11.724537    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:57:11.725035 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:11.725016    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:57:11.725752 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:11.725724    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 03:57:16.231066 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:57:16.231029    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:57:30.030192 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:30.030154    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"]
Oct 30 03:57:30.038905 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:30.038877    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ece13c-fzr2h"]
Oct 30 03:57:31.704634 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:31.704594    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1cb12d-fc8e-4ea3-a319-2678d2a1f249" path="/var/lib/kubelet/pods/4c1cb12d-fc8e-4ea3-a319-2678d2a1f249/volumes"
Oct 30 03:57:41.030470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:41.030375    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"]
Oct 30 03:57:41.039335 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:41.039306    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a4f986-plmg8"]
Oct 30 03:57:41.700570 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:41.700535    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f36106-0c2e-42ca-8654-15b263f94696" path="/var/lib/kubelet/pods/36f36106-0c2e-42ca-8654-15b263f94696/volumes"
Oct 30 03:57:46.230789 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:57:46.230751    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:57:52.884125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:52.884095    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:57:52.893969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:52.893940    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 03:57:54.028856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:54.028824    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"]
Oct 30 03:57:54.037866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:54.037837    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb83ed34-r47lw"]
Oct 30 03:57:55.702948 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:55.702890    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1399f95b-c193-4f1d-b5c3-38f19d40d1ab" path="/var/lib/kubelet/pods/1399f95b-c193-4f1d-b5c3-38f19d40d1ab/volumes"
Oct 30 03:57:56.215139 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:56.215100    4877 scope.go:122] "RemoveContainer" containerID="65994b9f200eee8416cc816b484bd2d0bbb71610166f4ffda843f75d10f8e949"
Oct 30 03:57:56.228789 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:56.228770    4877 scope.go:122] "RemoveContainer" containerID="4491a1da8a2405ea00389bc2b1b3de825addbd023efbee4a00a723976db79eb5"
Oct 30 03:57:56.240164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:57:56.240142    4877 scope.go:122] "RemoveContainer" containerID="d50c9bba711daaa87dd8796bb2d9ff40918fbcdf80adddd7e96a01083831fe1d"
Oct 30 03:58:04.032132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:04.032092    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"]
Oct 30 03:58:04.038240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:04.038213    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44943283-96szk"]
Oct 30 03:58:05.700356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:05.700321    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f231b8-3c9e-41d6-bd58-8cb578115096" path="/var/lib/kubelet/pods/86f231b8-3c9e-41d6-bd58-8cb578115096/volumes"
Oct 30 03:58:07.819914 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.819872    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"]
Oct 30 03:58:07.820863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820839    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy"
Oct 30 03:58:07.820863 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820861    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820875    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820884    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820892    4877 cpu_manager.go:414] "containerMap: removing container" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820899    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820907    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820914    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820932    4877 cpu_manager.go:414] "containerMap: removing container" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820941    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820972    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820978    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820992    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.820998    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821028    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821038    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821049    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" containerName="revoke-keys"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821056    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" containerName="revoke-keys"
Oct 30 03:58:07.821073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821076    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821083    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821102    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821110    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821128    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821135    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821161    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821169    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821344    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="198b1ec6-ac60-49d8-bda5-581bbb36c219" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821362    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821373    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821386    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821396    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" containerName="revoke-keys"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821410    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821437    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13a6e55-0c25-48e7-a08c-765c3c57ef77" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821450    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b99d5ad-c664-4fae-b8f8-ea76535a03c5" containerName="payload-pre-processing"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821464    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46386d6-80fd-40a9-aff1-1db047cfb8b9" containerName="istio-proxy"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821473    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b372d9-e957-4c98-ad09-a114e9c0e387" containerName="payload-processing"
Oct 30 03:58:07.821979 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.821481    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68516de-5dcd-4571-8208-937bddb2108c" containerName="istio-proxy"
Oct 30 03:58:07.825857 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.825834    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:07.844920 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.844895    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"]
Oct 30 03:58:07.848107 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.848081    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:58:07.848257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.848237    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:58:07.957334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.957299    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:07.957560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.957435    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcchw\" (UniqueName: \"kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:07.957560 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:07.957493    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.059019 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.058985    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.059226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.059057    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcchw\" (UniqueName: \"kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.059226 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.059095    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.059992 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.059961    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.061460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.061438    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.076651 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.076594    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcchw\" (UniqueName: \"kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw\") pod \"maas-api-revoke-sub-f03ca44c-bg29k\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.141271 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.141240    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:08.452588 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:58:08.452543    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadb0c55_dc64_42c6_b82f_99005c95ab47.slice/crio-ac1f64d1ef12572009f590a6aa6e98e6592252f9137a99d8d359a7ea7aca463b WatchSource:0}: Error finding container ac1f64d1ef12572009f590a6aa6e98e6592252f9137a99d8d359a7ea7aca463b: Status 404 returned error can't find the container with id ac1f64d1ef12572009f590a6aa6e98e6592252f9137a99d8d359a7ea7aca463b
Oct 30 03:58:08.456198 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:08.456164    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"]
Oct 30 03:58:09.032278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:09.032207    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k" event={"ID":"dadb0c55-dc64-42c6-b82f-99005c95ab47","Type":"ContainerStarted","Data":"c3aa580b6fde3ae89f00288f0381a3a34f22e2af6b30b95500936cb3af828c53"}
Oct 30 03:58:09.032768 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:09.032340    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k" event={"ID":"dadb0c55-dc64-42c6-b82f-99005c95ab47","Type":"ContainerStarted","Data":"ac1f64d1ef12572009f590a6aa6e98e6592252f9137a99d8d359a7ea7aca463b"}
Oct 30 03:58:09.075218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:09.074378    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k" podStartSLOduration=2.07435832 podStartE2EDuration="2.07435832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:09.067127158 +0000 UTC m=+3317.496016598" watchObservedRunningTime="2026-10-30 03:58:09.07435832 +0000 UTC m=+3317.503247741"
Oct 30 03:58:10.043958 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:10.043926    4877 generic.go:356] "Generic (PLEG): container finished" podID="dadb0c55-dc64-42c6-b82f-99005c95ab47" containerID="c3aa580b6fde3ae89f00288f0381a3a34f22e2af6b30b95500936cb3af828c53" exitCode=0
Oct 30 03:58:10.044368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:10.043995    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k" event={"ID":"dadb0c55-dc64-42c6-b82f-99005c95ab47","Type":"ContainerDied","Data":"c3aa580b6fde3ae89f00288f0381a3a34f22e2af6b30b95500936cb3af828c53"}
Oct 30 03:58:11.244269 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.244242    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:11.424613 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.424531    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca\") pod \"dadb0c55-dc64-42c6-b82f-99005c95ab47\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") "
Oct 30 03:58:11.424885 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.424869    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca\") pod \"dadb0c55-dc64-42c6-b82f-99005c95ab47\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") "
Oct 30 03:58:11.424938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.424909    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw\" (UniqueName: \"kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw\") pod \"dadb0c55-dc64-42c6-b82f-99005c95ab47\" (UID: \"dadb0c55-dc64-42c6-b82f-99005c95ab47\") "
Oct 30 03:58:11.424938 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.424918    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca" pod "dadb0c55-dc64-42c6-b82f-99005c95ab47" (UID: "dadb0c55-dc64-42c6-b82f-99005c95ab47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:11.425614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.425591    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.426941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.426909    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca" pod "dadb0c55-dc64-42c6-b82f-99005c95ab47" (UID: "dadb0c55-dc64-42c6-b82f-99005c95ab47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:11.427470 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.427417    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw" pod "dadb0c55-dc64-42c6-b82f-99005c95ab47" (UID: "dadb0c55-dc64-42c6-b82f-99005c95ab47"). InnerVolumeSpecName "kube-api-access-gcchw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:11.527236 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.527195    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadb0c55-dc64-42c6-b82f-99005c95ab47-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:11.527409 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:11.527245    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcchw\" (UniqueName: \"kubernetes.io/projected/dadb0c55-dc64-42c6-b82f-99005c95ab47-kube-api-access-gcchw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:12.064235 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:12.064198    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k" event={"ID":"dadb0c55-dc64-42c6-b82f-99005c95ab47","Type":"ContainerDied","Data":"ac1f64d1ef12572009f590a6aa6e98e6592252f9137a99d8d359a7ea7aca463b"}
Oct 30 03:58:12.064455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:12.064240    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"
Oct 30 03:58:12.064455 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:12.064250    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac1f64d1ef12572009f590a6aa6e98e6592252f9137a99d8d359a7ea7aca463b"
Oct 30 03:58:16.231823 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:58:16.231774    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:58:26.958932 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.958893    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"]
Oct 30 03:58:26.959727 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.959711    4877 cpu_manager.go:414] "containerMap: removing container" podUID="dadb0c55-dc64-42c6-b82f-99005c95ab47" containerName="revoke-keys"
Oct 30 03:58:26.959777 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.959728    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadb0c55-dc64-42c6-b82f-99005c95ab47" containerName="revoke-keys"
Oct 30 03:58:26.959897 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.959887    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63b3dd3-9d77-4f33-a4a7-024bf6053f4b" containerName="istio-proxy"
Oct 30 03:58:26.959935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.959899    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1620d8-851c-4b2e-bacf-be013a1cb957" containerName="istio-proxy"
Oct 30 03:58:26.959935 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.959913    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadb0c55-dc64-42c6-b82f-99005c95ab47" containerName="revoke-keys"
Oct 30 03:58:26.963835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.963817    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:26.981886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.981854    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"]
Oct 30 03:58:26.985221 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.985198    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:58:26.985346 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:26.985287    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:58:27.025655 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.025621    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.025834 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.025769    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdkh\" (UniqueName: \"kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.025888 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.025868    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.127209 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.127145    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.127395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.127248    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdkh\" (UniqueName: \"kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.127395 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.127290    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.127994 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.127967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.129811 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.129783    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.144058 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.144038    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdkh\" (UniqueName: \"kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh\") pod \"maas-api-revoke-sub-75a701a1-z8sgp\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.278548 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.278515    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:27.593092 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:58:27.593060    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30dd828_bfb1_4404_b2c3_0cd07375c56c.slice/crio-31c928671a736084c8eee9747a258b6df0f8e7318d2d13873617356a74c896e3 WatchSource:0}: Error finding container 31c928671a736084c8eee9747a258b6df0f8e7318d2d13873617356a74c896e3: Status 404 returned error can't find the container with id 31c928671a736084c8eee9747a258b6df0f8e7318d2d13873617356a74c896e3
Oct 30 03:58:27.595882 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:27.595852    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"]
Oct 30 03:58:28.273371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:28.273331    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp" event={"ID":"d30dd828-bfb1-4404-b2c3-0cd07375c56c","Type":"ContainerStarted","Data":"47662e0f70d09a5b0bfaaf11e42fa4fe95c49646f1e7e30441c2eb69090c5fd1"}
Oct 30 03:58:28.273866 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:28.273399    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp" event={"ID":"d30dd828-bfb1-4404-b2c3-0cd07375c56c","Type":"ContainerStarted","Data":"31c928671a736084c8eee9747a258b6df0f8e7318d2d13873617356a74c896e3"}
Oct 30 03:58:28.320106 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:28.320059    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp" podStartSLOduration=2.32004501 podStartE2EDuration="2.32004501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:28.30816017 +0000 UTC m=+3336.737049592" watchObservedRunningTime="2026-10-30 03:58:28.32004501 +0000 UTC m=+3336.748934425"
Oct 30 03:58:29.283624 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:29.283585    4877 generic.go:356] "Generic (PLEG): container finished" podID="d30dd828-bfb1-4404-b2c3-0cd07375c56c" containerID="47662e0f70d09a5b0bfaaf11e42fa4fe95c49646f1e7e30441c2eb69090c5fd1" exitCode=0
Oct 30 03:58:29.284023 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:29.283666    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp" event={"ID":"d30dd828-bfb1-4404-b2c3-0cd07375c56c","Type":"ContainerDied","Data":"47662e0f70d09a5b0bfaaf11e42fa4fe95c49646f1e7e30441c2eb69090c5fd1"}
Oct 30 03:58:30.488218 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.488190    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:30.618896 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.618798    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh\" (UniqueName: \"kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh\") pod \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") "
Oct 30 03:58:30.619080 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.618946    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca\") pod \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") "
Oct 30 03:58:30.619161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.619096    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca\") pod \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\" (UID: \"d30dd828-bfb1-4404-b2c3-0cd07375c56c\") "
Oct 30 03:58:30.619509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.619480    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca" pod "d30dd828-bfb1-4404-b2c3-0cd07375c56c" (UID: "d30dd828-bfb1-4404-b2c3-0cd07375c56c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:30.619743 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.619710    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:30.621125 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.621102    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca" pod "d30dd828-bfb1-4404-b2c3-0cd07375c56c" (UID: "d30dd828-bfb1-4404-b2c3-0cd07375c56c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:30.621213 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.621131    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh" pod "d30dd828-bfb1-4404-b2c3-0cd07375c56c" (UID: "d30dd828-bfb1-4404-b2c3-0cd07375c56c"). InnerVolumeSpecName "kube-api-access-stdkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:30.721081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.721039    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stdkh\" (UniqueName: \"kubernetes.io/projected/d30dd828-bfb1-4404-b2c3-0cd07375c56c-kube-api-access-stdkh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:30.721081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:30.721077    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30dd828-bfb1-4404-b2c3-0cd07375c56c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:31.304397 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:31.304362    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"
Oct 30 03:58:31.304610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:31.304389    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp" event={"ID":"d30dd828-bfb1-4404-b2c3-0cd07375c56c","Type":"ContainerDied","Data":"31c928671a736084c8eee9747a258b6df0f8e7318d2d13873617356a74c896e3"}
Oct 30 03:58:31.304610 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:31.304493    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c928671a736084c8eee9747a258b6df0f8e7318d2d13873617356a74c896e3"
Oct 30 03:58:38.416614 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.416564    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"]
Oct 30 03:58:38.417740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.417714    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d30dd828-bfb1-4404-b2c3-0cd07375c56c" containerName="revoke-keys"
Oct 30 03:58:38.417740 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.417738    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30dd828-bfb1-4404-b2c3-0cd07375c56c" containerName="revoke-keys"
Oct 30 03:58:38.417969 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.417953    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30dd828-bfb1-4404-b2c3-0cd07375c56c" containerName="revoke-keys"
Oct 30 03:58:38.422683 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.422660    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.439041 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.439018    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"]
Oct 30 03:58:38.442661 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.442639    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:58:38.442760 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.442711    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:58:38.464991 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.464952    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874xg\" (UniqueName: \"kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.465489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.465352    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.465489 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.465405    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.592750 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.587853    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.594547 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.594517    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.594900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.594862    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874xg\" (UniqueName: \"kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.595508 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.595465    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.600791 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.600766    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.612830 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.612806    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874xg\" (UniqueName: \"kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg\") pod \"maas-api-revoke-sub-74884c5b-7c2xx\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:38.738804 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:38.738734    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:39.056086 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:58:39.056051    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace82714_9e24_49c7_a293_6fa9c027f79d.slice/crio-a2ea704e9cd177c519bf7c3c82816a87477de5c93011eb59f6a2c5878883671f WatchSource:0}: Error finding container a2ea704e9cd177c519bf7c3c82816a87477de5c93011eb59f6a2c5878883671f: Status 404 returned error can't find the container with id a2ea704e9cd177c519bf7c3c82816a87477de5c93011eb59f6a2c5878883671f
Oct 30 03:58:39.058004 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:39.057974    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"]
Oct 30 03:58:39.383546 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:39.383457    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx" event={"ID":"ace82714-9e24-49c7-a293-6fa9c027f79d","Type":"ContainerStarted","Data":"a2ea704e9cd177c519bf7c3c82816a87477de5c93011eb59f6a2c5878883671f"}
Oct 30 03:58:40.397365 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:40.397277    4877 generic.go:356] "Generic (PLEG): container finished" podID="ace82714-9e24-49c7-a293-6fa9c027f79d" containerID="e8029b919c9c251ae245b77f682992206e4f20b32c43c9931db537f21d4d3b7d" exitCode=0
Oct 30 03:58:40.397831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:40.397353    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx" event={"ID":"ace82714-9e24-49c7-a293-6fa9c027f79d","Type":"ContainerDied","Data":"e8029b919c9c251ae245b77f682992206e4f20b32c43c9931db537f21d4d3b7d"}
Oct 30 03:58:41.121877 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:41.121830    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"]
Oct 30 03:58:41.122134 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:41.122104    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy" containerID="cri-o://82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4" gracePeriod=30
Oct 30 03:58:41.132356 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:41.132322    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"]
Oct 30 03:58:41.132607 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:41.132577    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy" containerID="cri-o://0cf554f7f48a9fda54b22251074b869fcd484c17a81f8a00572611066d20e905" gracePeriod=30
Oct 30 03:58:41.955446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:41.955398    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:42.032040 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.032002    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca\") pod \"ace82714-9e24-49c7-a293-6fa9c027f79d\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") "
Oct 30 03:58:42.032453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.032409    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg\" (UniqueName: \"kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg\") pod \"ace82714-9e24-49c7-a293-6fa9c027f79d\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") "
Oct 30 03:58:42.032656 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.032636    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca\") pod \"ace82714-9e24-49c7-a293-6fa9c027f79d\" (UID: \"ace82714-9e24-49c7-a293-6fa9c027f79d\") "
Oct 30 03:58:42.068446 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.064710    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca" pod "ace82714-9e24-49c7-a293-6fa9c027f79d" (UID: "ace82714-9e24-49c7-a293-6fa9c027f79d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:42.092649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.074961    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca" pod "ace82714-9e24-49c7-a293-6fa9c027f79d" (UID: "ace82714-9e24-49c7-a293-6fa9c027f79d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:42.092649 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.075088    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg" pod "ace82714-9e24-49c7-a293-6fa9c027f79d" (UID: "ace82714-9e24-49c7-a293-6fa9c027f79d"). InnerVolumeSpecName "kube-api-access-874xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:42.146066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.144492    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:42.146066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.144524    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874xg\" (UniqueName: \"kubernetes.io/projected/ace82714-9e24-49c7-a293-6fa9c027f79d-kube-api-access-874xg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:42.146066 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.144540    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace82714-9e24-49c7-a293-6fa9c027f79d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:42.450709 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.450614    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx" event={"ID":"ace82714-9e24-49c7-a293-6fa9c027f79d","Type":"ContainerDied","Data":"a2ea704e9cd177c519bf7c3c82816a87477de5c93011eb59f6a2c5878883671f"}
Oct 30 03:58:42.450970 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.450947    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2ea704e9cd177c519bf7c3c82816a87477de5c93011eb59f6a2c5878883671f"
Oct 30 03:58:42.451149 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:42.451135    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"
Oct 30 03:58:46.233245 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:58:46.233198    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:58:46.452200 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.452171    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:58:46.521310 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.521275    4877 generic.go:356] "Generic (PLEG): container finished" podID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerID="0cf554f7f48a9fda54b22251074b869fcd484c17a81f8a00572611066d20e905" exitCode=0
Oct 30 03:58:46.521466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.521365    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" event={"ID":"de8df1dd-105c-4207-bcaa-596b7f0351e1","Type":"ContainerDied","Data":"0cf554f7f48a9fda54b22251074b869fcd484c17a81f8a00572611066d20e905"}
Oct 30 03:58:46.521466 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.521442    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j" event={"ID":"de8df1dd-105c-4207-bcaa-596b7f0351e1","Type":"ContainerDied","Data":"862755d1aee842f19e13528850d3fdb0400584399643c299bad71a082ad063de"}
Oct 30 03:58:46.521601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.521477    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862755d1aee842f19e13528850d3fdb0400584399643c299bad71a082ad063de"
Oct 30 03:58:46.527453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.523485    4877 generic.go:356] "Generic (PLEG): container finished" podID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerID="82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4" exitCode=0
Oct 30 03:58:46.527453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.523646    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" event={"ID":"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1","Type":"ContainerDied","Data":"82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4"}
Oct 30 03:58:46.527453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.523691    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw" event={"ID":"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1","Type":"ContainerDied","Data":"8cc8a2e13de5424de0d8ea28a97d034e65abf092c25d4160333d9a5652f3be88"}
Oct 30 03:58:46.527453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.523727    4877 scope.go:122] "RemoveContainer" containerID="82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4"
Oct 30 03:58:46.527453 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.523897    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"
Oct 30 03:58:46.536086 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.536056    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:58:46.543682 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.543652    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.543862 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.543842    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.543941 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.543926    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544009 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.543977    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544073 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544025    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544062    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.544132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544077    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544140    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544149    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.544257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544188    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544257 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544236    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo\") pod \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\" (UID: \"fe9a8a23-ed54-462d-bafa-2aa064a9d7c1\") "
Oct 30 03:58:46.544494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544339    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.544494 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544412    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:46.544880 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.544853    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.545132 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.545110    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.545207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.545137    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.545207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.545157    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.545207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.545170    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.545207 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.545183    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.546900 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.546877    4877 scope.go:122] "RemoveContainer" containerID="82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4"
Oct 30 03:58:46.547076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.547054    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:58:46.547289 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:58:46.547262    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4\": container with ID starting with 82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4 not found: ID does not exist" containerID="82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4"
Oct 30 03:58:46.547371 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.547301    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4"} err="failed to get container status \"82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4\": rpc error: code = NotFound desc = could not find container \"82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4\": container with ID starting with 82afe2ab47c33b4d2cab370ede74b277afb02ac31b387fc7cef46cd21c3a71d4 not found: ID does not exist"
Oct 30 03:58:46.548094 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.548060    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.549603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.549566    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "kube-api-access-fm786". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:46.549712 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.549667    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token" pod "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" (UID: "fe9a8a23-ed54-462d-bafa-2aa064a9d7c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:46.645869 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.645838    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.645893    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.645974    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646018    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646067 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646046    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646203    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646333 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646274    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646469 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646446    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646530 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646475    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.646597 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646554    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket\") pod \"de8df1dd-105c-4207-bcaa-596b7f0351e1\" (UID: \"de8df1dd-105c-4207-bcaa-596b7f0351e1\") "
Oct 30 03:58:46.646764 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646738    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:46.646856 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646827    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.646953 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.646920    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.647062 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647014    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647608    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de8df1dd-105c-4207-bcaa-596b7f0351e1-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647638    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647658    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647675    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647689    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647705    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647720    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm786\" (UniqueName: \"kubernetes.io/projected/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-kube-api-access-fm786\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647733    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.647771 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.647746    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.648665 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.648637    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:46.648784 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.648722    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:58:46.649098 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.649078    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:46.649268 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.649250    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9" pod "de8df1dd-105c-4207-bcaa-596b7f0351e1" (UID: "de8df1dd-105c-4207-bcaa-596b7f0351e1"). InnerVolumeSpecName "kube-api-access-zxfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:46.757156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.754660    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfx9\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-kube-api-access-zxfx9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.757156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.754701    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.757156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.754717    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.757156 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.754730    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de8df1dd-105c-4207-bcaa-596b7f0351e1-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:58:46.868568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.868536    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"]
Oct 30 03:58:46.873505 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:46.873480    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-x4kxw"]
Oct 30 03:58:47.543152 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:47.543114    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"
Oct 30 03:58:47.604590 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:47.604551    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"]
Oct 30 03:58:47.610955 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:47.610923    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-zx49j"]
Oct 30 03:58:47.701558 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:47.701527    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" path="/var/lib/kubelet/pods/de8df1dd-105c-4207-bcaa-596b7f0351e1/volumes"
Oct 30 03:58:47.702051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:47.702037    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" path="/var/lib/kubelet/pods/fe9a8a23-ed54-462d-bafa-2aa064a9d7c1/volumes"
Oct 30 03:58:56.299612 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:56.299574    4877 scope.go:122] "RemoveContainer" containerID="6ac146966556d3e49208ffd9e580ea46c4374f537320532b76d1dcf50bf6dabd"
Oct 30 03:58:56.314240 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:56.314218    4877 scope.go:122] "RemoveContainer" containerID="70dd4dab51d0523ef5d87ae4c128856f6f2df6c909922780628326dc8daca419"
Oct 30 03:58:56.325639 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:58:56.325613    4877 scope.go:122] "RemoveContainer" containerID="b4ed20ec24f3b527628cc6c8b10852c87d5ef0ff1255fb5d8f25cb6912abe3f7"
Oct 30 03:59:16.234902 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:59:16.234804    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:59:33.407817 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:33.407775    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"]
Oct 30 03:59:33.408718 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:33.408677    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy" containerID="cri-o://13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5" gracePeriod=30
Oct 30 03:59:38.746701 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.746675    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:59:38.848835 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.848799    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.848900    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.848954    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849249 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849226    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849450 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849375    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849454    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849580 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849283    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:38.849721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849473    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:38.849721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849590    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849665    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.849721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849518    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:38.849721 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.849695    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token\") pod \"e3660296-8703-4234-84cd-c8b96142c79f\" (UID: \"e3660296-8703-4234-84cd-c8b96142c79f\") "
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850010    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850685    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850933    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850952    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850967    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850980    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3660296-8703-4234-84cd-c8b96142c79f-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.851044 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.850992    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.853868 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.853846    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:38.857913 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.857879    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:59:38.858182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.858164    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "kube-api-access-hmsrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:38.859278 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.859244    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token" pod "e3660296-8703-4234-84cd-c8b96142c79f" (UID: "e3660296-8703-4234-84cd-c8b96142c79f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:38.876164 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.876141    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"]
Oct 30 03:59:38.876990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.876970    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy"
Oct 30 03:59:38.876990 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.876988    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877011    4877 cpu_manager.go:414] "containerMap: removing container" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877018    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877060    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877067    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877080    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877088    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877129    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ace82714-9e24-49c7-a293-6fa9c027f79d" containerName="revoke-keys"
Oct 30 03:59:38.877161 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877137    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace82714-9e24-49c7-a293-6fa9c027f79d" containerName="revoke-keys"
Oct 30 03:59:38.877628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877353    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9a8a23-ed54-462d-bafa-2aa064a9d7c1" containerName="istio-proxy"
Oct 30 03:59:38.877628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877367    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy"
Oct 30 03:59:38.877628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877381    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace82714-9e24-49c7-a293-6fa9c027f79d" containerName="revoke-keys"
Oct 30 03:59:38.877628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877399    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3660296-8703-4234-84cd-c8b96142c79f" containerName="istio-proxy"
Oct 30 03:59:38.877628 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.877410    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="de8df1dd-105c-4207-bcaa-596b7f0351e1" containerName="istio-proxy"
Oct 30 03:59:38.895076 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.895041    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:38.897798 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.897769    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"]
Oct 30 03:59:38.907592 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.907559    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:59:38.907723 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.907669    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:59:38.952338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.952297    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3660296-8703-4234-84cd-c8b96142c79f-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.952338 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.952331    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsrd\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-kube-api-access-hmsrd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.952605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.952346    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3660296-8703-4234-84cd-c8b96142c79f-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:38.952605 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:38.952360    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3660296-8703-4234-84cd-c8b96142c79f-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:39.053864 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.053826    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.054081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.054000    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.054081 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.054033    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbkl\" (UniqueName: \"kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.155886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.155848    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.155886 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.155887    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbkl\" (UniqueName: \"kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.156184 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.156152    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.156601 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.156582    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.158611 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.158590    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.174700 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.174681    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbkl\" (UniqueName: \"kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl\") pod \"maas-api-revoke-sub-c1194c82-2djkl\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.216562 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.216526    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:39.363275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.362448    4877 generic.go:356] "Generic (PLEG): container finished" podID="e3660296-8703-4234-84cd-c8b96142c79f" containerID="13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5" exitCode=0
Oct 30 03:59:39.363275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.362826    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" event={"ID":"e3660296-8703-4234-84cd-c8b96142c79f","Type":"ContainerDied","Data":"13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5"}
Oct 30 03:59:39.363275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.362905    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww" event={"ID":"e3660296-8703-4234-84cd-c8b96142c79f","Type":"ContainerDied","Data":"7fecc357e08f55d8a72413bd93b4c30275db87c0a6729d87f0ac4dc6717fb3e8"}
Oct 30 03:59:39.363275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.362939    4877 scope.go:122] "RemoveContainer" containerID="13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5"
Oct 30 03:59:39.363275 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.363182    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"
Oct 30 03:59:39.385575 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.385552    4877 scope.go:122] "RemoveContainer" containerID="1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"
Oct 30 03:59:39.401600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.399148    4877 scope.go:122] "RemoveContainer" containerID="13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5"
Oct 30 03:59:39.401600 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:59:39.399523    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5\": container with ID starting with 13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5 not found: ID does not exist" containerID="13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5"
Oct 30 03:59:39.401600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.399559    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5"} err="failed to get container status \"13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5\": rpc error: code = NotFound desc = could not find container \"13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5\": container with ID starting with 13ad04605308070287b7db0978301a070d445da691dc71a55b073f93d3d206c5 not found: ID does not exist"
Oct 30 03:59:39.401600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.399583    4877 scope.go:122] "RemoveContainer" containerID="1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"
Oct 30 03:59:39.401600 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:59:39.399857    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788\": container with ID starting with 1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788 not found: ID does not exist" containerID="1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"
Oct 30 03:59:39.401600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.399884    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788"} err="failed to get container status \"1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788\": rpc error: code = NotFound desc = could not find container \"1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788\": container with ID starting with 1698f018cb1eca8d9a3fa1594cb6a14704b52747fddc6e450cd0c4fc385ce788 not found: ID does not exist"
Oct 30 03:59:39.427603 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.427567    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"]
Oct 30 03:59:39.433258 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.432345    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3cfed7-openshift-default-d98ff4c9f-b7vww"]
Oct 30 03:59:39.588657 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:59:39.588585    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dc673d_8c55_466a_af7f_b6f6033c7bab.slice/crio-cb5f5ded050042dd9df9766815fc146efb0771f82e4273f438be7c96360c981c WatchSource:0}: Error finding container cb5f5ded050042dd9df9766815fc146efb0771f82e4273f438be7c96360c981c: Status 404 returned error can't find the container with id cb5f5ded050042dd9df9766815fc146efb0771f82e4273f438be7c96360c981c
Oct 30 03:59:39.589319 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.589198    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"]
Oct 30 03:59:39.701051 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.701017    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3660296-8703-4234-84cd-c8b96142c79f" path="/var/lib/kubelet/pods/e3660296-8703-4234-84cd-c8b96142c79f/volumes"
Oct 30 03:59:39.930688 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:39.930655    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"]
Oct 30 03:59:40.383824 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:40.383783    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl" event={"ID":"57dc673d-8c55-466a-af7f-b6f6033c7bab","Type":"ContainerStarted","Data":"f9b3ab5a69cc73b7de8fccf8cb79a5ad728e4ddbe9f3c5e295e0f6c85101ae04"}
Oct 30 03:59:40.384026 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:40.383849    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl" event={"ID":"57dc673d-8c55-466a-af7f-b6f6033c7bab","Type":"ContainerStarted","Data":"cb5f5ded050042dd9df9766815fc146efb0771f82e4273f438be7c96360c981c"}
Oct 30 03:59:40.420904 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:40.420845    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl" podStartSLOduration=2.420831126 podStartE2EDuration="2.420831126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:40.419098613 +0000 UTC m=+3408.847988058" watchObservedRunningTime="2026-10-30 03:59:40.420831126 +0000 UTC m=+3408.849720544"
Oct 30 03:59:41.395815 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:41.395774    4877 generic.go:356] "Generic (PLEG): container finished" podID="57dc673d-8c55-466a-af7f-b6f6033c7bab" containerID="f9b3ab5a69cc73b7de8fccf8cb79a5ad728e4ddbe9f3c5e295e0f6c85101ae04" exitCode=0
Oct 30 03:59:41.396290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:41.395843    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl" event={"ID":"57dc673d-8c55-466a-af7f-b6f6033c7bab","Type":"ContainerDied","Data":"f9b3ab5a69cc73b7de8fccf8cb79a5ad728e4ddbe9f3c5e295e0f6c85101ae04"}
Oct 30 03:59:42.651437 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.651396    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:42.840185 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.840154    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl\" (UniqueName: \"kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl\") pod \"57dc673d-8c55-466a-af7f-b6f6033c7bab\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") "
Oct 30 03:59:42.840374 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.840339    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca\") pod \"57dc673d-8c55-466a-af7f-b6f6033c7bab\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") "
Oct 30 03:59:42.840482 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.840461    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca\") pod \"57dc673d-8c55-466a-af7f-b6f6033c7bab\" (UID: \"57dc673d-8c55-466a-af7f-b6f6033c7bab\") "
Oct 30 03:59:42.840774 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.840750    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca" pod "57dc673d-8c55-466a-af7f-b6f6033c7bab" (UID: "57dc673d-8c55-466a-af7f-b6f6033c7bab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:42.841461 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.841416    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:42.842519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.842497    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl" pod "57dc673d-8c55-466a-af7f-b6f6033c7bab" (UID: "57dc673d-8c55-466a-af7f-b6f6033c7bab"). InnerVolumeSpecName "kube-api-access-spbkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:42.842600 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.842529    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca" pod "57dc673d-8c55-466a-af7f-b6f6033c7bab" (UID: "57dc673d-8c55-466a-af7f-b6f6033c7bab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:42.943827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.943788    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbkl\" (UniqueName: \"kubernetes.io/projected/57dc673d-8c55-466a-af7f-b6f6033c7bab-kube-api-access-spbkl\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:42.943827 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:42.943830    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57dc673d-8c55-466a-af7f-b6f6033c7bab-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:43.423874 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:43.423843    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"
Oct 30 03:59:43.424087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:43.423873    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl" event={"ID":"57dc673d-8c55-466a-af7f-b6f6033c7bab","Type":"ContainerDied","Data":"cb5f5ded050042dd9df9766815fc146efb0771f82e4273f438be7c96360c981c"}
Oct 30 03:59:43.424087 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:43.423922    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb5f5ded050042dd9df9766815fc146efb0771f82e4273f438be7c96360c981c"
Oct 30 03:59:44.931568 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:44.931523    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp" podUID="d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" containerName="payload-processing" containerID="cri-o://1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e" gracePeriod=25
Oct 30 03:59:45.234865 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.234838    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:59:45.302519 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.302411    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume\") pod \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") "
Oct 30 03:59:45.306165 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.302975    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume" pod "d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" (UID: "d1bc4dcf-f2b4-461a-9cda-5430dc76e21a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:45.306500 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.306469    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j\" (UniqueName: \"kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j\") pod \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\" (UID: \"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a\") "
Oct 30 03:59:45.307368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.307340    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:45.311456 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.309531    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j" pod "d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" (UID: "d1bc4dcf-f2b4-461a-9cda-5430dc76e21a"). InnerVolumeSpecName "kube-api-access-wzx6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:45.408890 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.408846    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzx6j\" (UniqueName: \"kubernetes.io/projected/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a-kube-api-access-wzx6j\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:45.454497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.448451    4877 generic.go:356] "Generic (PLEG): container finished" podID="d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" containerID="1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e" exitCode=0
Oct 30 03:59:45.454497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.448706    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp" event={"ID":"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a","Type":"ContainerDied","Data":"1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e"}
Oct 30 03:59:45.454497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.448806    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp" event={"ID":"d1bc4dcf-f2b4-461a-9cda-5430dc76e21a","Type":"ContainerDied","Data":"563bf196e59e620183a61c3d131eb960a3034d10eac02a5293028c5a37000f0f"}
Oct 30 03:59:45.454497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.448841    4877 scope.go:122] "RemoveContainer" containerID="1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e"
Oct 30 03:59:45.454497 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.449065    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"
Oct 30 03:59:45.535625 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.535580    4877 scope.go:122] "RemoveContainer" containerID="1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e"
Oct 30 03:59:45.535999 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:59:45.535973    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e\": container with ID starting with 1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e not found: ID does not exist" containerID="1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e"
Oct 30 03:59:45.536090 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.536007    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e"} err="failed to get container status \"1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e\": rpc error: code = NotFound desc = could not find container \"1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e\": container with ID starting with 1c03648534bf1700412ac67794cc88d91fe2bd91933981db0e67794e7c72b37e not found: ID does not exist"
Oct 30 03:59:45.556111 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.556078    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"]
Oct 30 03:59:45.561736 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.561708    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9e0d09-6d59559648-rnwsp"]
Oct 30 03:59:45.700154 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:45.700113    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" path="/var/lib/kubelet/pods/d1bc4dcf-f2b4-461a-9cda-5430dc76e21a/volumes"
Oct 30 03:59:46.231623 ip-10-0-135-226 kubenswrapper[4877]: E1030 03:59:46.231580    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 03:59:50.295435 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.295384    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"]
Oct 30 03:59:50.296385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.296362    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" containerName="payload-processing"
Oct 30 03:59:50.296385 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.296381    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" containerName="payload-processing"
Oct 30 03:59:50.296591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.296391    4877 cpu_manager.go:414] "containerMap: removing container" podUID="57dc673d-8c55-466a-af7f-b6f6033c7bab" containerName="revoke-keys"
Oct 30 03:59:50.296591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.296398    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dc673d-8c55-466a-af7f-b6f6033c7bab" containerName="revoke-keys"
Oct 30 03:59:50.296591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.296561    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1bc4dcf-f2b4-461a-9cda-5430dc76e21a" containerName="payload-processing"
Oct 30 03:59:50.296591 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.296572    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dc673d-8c55-466a-af7f-b6f6033c7bab" containerName="revoke-keys"
Oct 30 03:59:50.300290 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.300269    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.333287 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.333250    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"]
Oct 30 03:59:50.360046 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.360013    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 03:59:50.360238 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.360211    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 03:59:50.400978 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.400929    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfdw\" (UniqueName: \"kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.401195 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.401056    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.401284 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.401266    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.502509 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.502468    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.502732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.502589    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfdw\" (UniqueName: \"kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.502732 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.502639    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.503232 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.503212    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.505573 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.505487    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.519368 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.519342    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfdw\" (UniqueName: \"kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw\") pod \"maas-api-revoke-sub-99e5af3b-nvnnm\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.620929 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.620840    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:50.935182 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:50.935150    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"]
Oct 30 03:59:50.936493 ip-10-0-135-226 kubenswrapper[4877]: W1030 03:59:50.936467    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837d41e7_4a80_4017_9fb1_e5ca7b612a0d.slice/crio-80f35ce23d38da024f9ad4dc3d684fc9e15918fe862c535626e3b2709cdd3ad2 WatchSource:0}: Error finding container 80f35ce23d38da024f9ad4dc3d684fc9e15918fe862c535626e3b2709cdd3ad2: Status 404 returned error can't find the container with id 80f35ce23d38da024f9ad4dc3d684fc9e15918fe862c535626e3b2709cdd3ad2
Oct 30 03:59:51.515675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:51.515586    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm" event={"ID":"837d41e7-4a80-4017-9fb1-e5ca7b612a0d","Type":"ContainerStarted","Data":"53006d6cfe639bd060f3c8ec5d111c6d8367056dd93e0b9959945779afb9e521"}
Oct 30 03:59:51.515675 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:51.515650    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm" event={"ID":"837d41e7-4a80-4017-9fb1-e5ca7b612a0d","Type":"ContainerStarted","Data":"80f35ce23d38da024f9ad4dc3d684fc9e15918fe862c535626e3b2709cdd3ad2"}
Oct 30 03:59:51.560115 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:51.560055    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm" podStartSLOduration=1.560040155 podStartE2EDuration="1.560040155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:51.548664982 +0000 UTC m=+3419.977554401" watchObservedRunningTime="2026-10-30 03:59:51.560040155 +0000 UTC m=+3419.988929573"
Oct 30 03:59:52.526398 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:52.526362    4877 generic.go:356] "Generic (PLEG): container finished" podID="837d41e7-4a80-4017-9fb1-e5ca7b612a0d" containerID="53006d6cfe639bd060f3c8ec5d111c6d8367056dd93e0b9959945779afb9e521" exitCode=0
Oct 30 03:59:52.526872 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:52.526456    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm" event={"ID":"837d41e7-4a80-4017-9fb1-e5ca7b612a0d","Type":"ContainerDied","Data":"53006d6cfe639bd060f3c8ec5d111c6d8367056dd93e0b9959945779afb9e521"}
Oct 30 03:59:53.723334 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.723305    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:53.852285 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.852194    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca\") pod \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") "
Oct 30 03:59:53.852460 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.852368    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca\") pod \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") "
Oct 30 03:59:53.852569 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.852470    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw\" (UniqueName: \"kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw\") pod \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\" (UID: \"837d41e7-4a80-4017-9fb1-e5ca7b612a0d\") "
Oct 30 03:59:53.852831 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.852801    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca" pod "837d41e7-4a80-4017-9fb1-e5ca7b612a0d" (UID: "837d41e7-4a80-4017-9fb1-e5ca7b612a0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:53.853330 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.853305    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:53.854853 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.854821    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw" pod "837d41e7-4a80-4017-9fb1-e5ca7b612a0d" (UID: "837d41e7-4a80-4017-9fb1-e5ca7b612a0d"). InnerVolumeSpecName "kube-api-access-nsfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:53.855104 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.855083    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca" pod "837d41e7-4a80-4017-9fb1-e5ca7b612a0d" (UID: "837d41e7-4a80-4017-9fb1-e5ca7b612a0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:53.954983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.954949    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:53.954983 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:53.954978    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsfdw\" (UniqueName: \"kubernetes.io/projected/837d41e7-4a80-4017-9fb1-e5ca7b612a0d-kube-api-access-nsfdw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 03:59:54.547730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:54.547691    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"
Oct 30 03:59:54.547730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:54.547713    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm" event={"ID":"837d41e7-4a80-4017-9fb1-e5ca7b612a0d","Type":"ContainerDied","Data":"80f35ce23d38da024f9ad4dc3d684fc9e15918fe862c535626e3b2709cdd3ad2"}
Oct 30 03:59:54.547997 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:54.547791    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80f35ce23d38da024f9ad4dc3d684fc9e15918fe862c535626e3b2709cdd3ad2"
Oct 30 03:59:56.381224 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:56.381190    4877 scope.go:122] "RemoveContainer" containerID="1aac5d258011616bbf99a664dccb86871ba9e83c11ca981dc7d8ede481c14bd5"
Oct 30 03:59:57.030435 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:57.030385    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"]
Oct 30 03:59:57.039730 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:57.039704    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18710234-v6qwg"]
Oct 30 03:59:57.703193 ip-10-0-135-226 kubenswrapper[4877]: I1030 03:59:57.703155    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec3120d-e1f0-4cff-a33b-3b1c11050e6e" path="/var/lib/kubelet/pods/3ec3120d-e1f0-4cff-a33b-3b1c11050e6e/volumes"
Oct 30 04:00:00.145307 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.145266    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"]
Oct 30 04:00:00.146827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.146797    4877 cpu_manager.go:414] "containerMap: removing container" podUID="837d41e7-4a80-4017-9fb1-e5ca7b612a0d" containerName="revoke-keys"
Oct 30 04:00:00.146827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.146823    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837d41e7-4a80-4017-9fb1-e5ca7b612a0d" containerName="revoke-keys"
Oct 30 04:00:00.147012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.146980    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="837d41e7-4a80-4017-9fb1-e5ca7b612a0d" containerName="revoke-keys"
Oct 30 04:00:00.152326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.152305    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"]
Oct 30 04:00:00.152452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.152415    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.169125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.169086    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"]
Oct 30 04:00:00.190408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.185215    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:00:00.190408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.186104    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.237242 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.237214    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"]
Oct 30 04:00:00.243296 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.243267    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.267114 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.259677    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"]
Oct 30 04:00:00.275950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.274057    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.282494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.277798    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4nq\" (UniqueName: \"kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq\") pod \"maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.282494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.277926    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.282494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.277988    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67sr\" (UniqueName: \"kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.282494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.278024    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.282494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.280868    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"]
Oct 30 04:00:00.282494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.282250    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.336601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.333211    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"]
Oct 30 04:00:00.336601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.333752    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.358018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.357983    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"]
Oct 30 04:00:00.362913 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.359543    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.382902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.382477    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.383052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.382907    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67sr\" (UniqueName: \"kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.383052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.382943    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.383052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383017    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.383229 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383123    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5bb\" (UniqueName: \"kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.383229 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383197    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rjt\" (UniqueName: \"kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.383343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383273    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4t5\" (UniqueName: \"kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.383402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383354    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.383484 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383410    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4nq\" (UniqueName: \"kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq\") pod \"maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.383544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383519    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.383857 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.383827    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.384265 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.384241    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.396781 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.396719    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"]
Oct 30 04:00:00.397163 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.397141    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.406804    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"]
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.406847    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"]
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.409361    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.412796    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"]
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.412833    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"]
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.412872    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"]
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.412971    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.416437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.415315    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"]
Oct 30 04:00:00.420036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.419034    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67sr\" (UniqueName: \"kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr\") pod \"maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.436203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.433218    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"]
Oct 30 04:00:00.436203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.433846    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4nq\" (UniqueName: \"kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq\") pod \"maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.436203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.435577    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.439631 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.439545    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"]
Oct 30 04:00:00.439769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.439715    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.445564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.445543    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"]
Oct 30 04:00:00.445702 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.445621    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.451509 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.451489    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"]
Oct 30 04:00:00.451825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.451712    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.457233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.457213    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"]
Oct 30 04:00:00.457365 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.457324    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.463078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.463053    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"]
Oct 30 04:00:00.463194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.463143    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.470206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.470184    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"]
Oct 30 04:00:00.470350 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.470329    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.475218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.475200    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"]
Oct 30 04:00:00.475341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.475306    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.484725 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.484700    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.484848 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.484736    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4t5\" (UniqueName: \"kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.485176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485155    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.485266 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485193    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.485266 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485223    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgxn\" (UniqueName: \"kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn\") pod \"maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.485266 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485251    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmq65\" (UniqueName: \"kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.485442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485379    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.485504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485448    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.485559 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485514    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.485618 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485573    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.485673 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485615    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kxw\" (UniqueName: \"kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.485673 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485657    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5jp\" (UniqueName: \"kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.485775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485713    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4rl\" (UniqueName: \"kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.485837 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485778    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.485837 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485812    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfpg\" (UniqueName: \"kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.485939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485851    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5bb\" (UniqueName: \"kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.485939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485925    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.486055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485938    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.486055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.485970    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rjt\" (UniqueName: \"kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.486628 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.486599    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.487020 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.487000    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.488950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.488927    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"]
Oct 30 04:00:00.489072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.488981    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"]
Oct 30 04:00:00.489072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.489010    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"]
Oct 30 04:00:00.489072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.489046    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.490158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.490134    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"]
Oct 30 04:00:00.497380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.497359    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"]
Oct 30 04:00:00.502642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.502589    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4t5\" (UniqueName: \"kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.504039 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.504012    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5bb\" (UniqueName: \"kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.506056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.504647    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"]
Oct 30 04:00:00.506056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.505093    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rjt\" (UniqueName: \"kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.510749 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.510730    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.512190 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.512171    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"]
Oct 30 04:00:00.519638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.519575    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"]
Oct 30 04:00:00.535059 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.535032    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"]
Oct 30 04:00:00.537329 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.537307    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:00.543256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.543239    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"]
Oct 30 04:00:00.550816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.550796    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"]
Oct 30 04:00:00.558832 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.558813    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"]
Oct 30 04:00:00.565003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.564985    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"]
Oct 30 04:00:00.570548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.570527    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"]
Oct 30 04:00:00.576808 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.576753    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"]
Oct 30 04:00:00.580092 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.579851    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:00.582098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.582075    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"]
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587249    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7r9t\" (UniqueName: \"kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587323    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587366    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587406    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587454    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587480    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587510    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvhr\" (UniqueName: \"kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr\") pod \"maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587548    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.587634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587591    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kxw\" (UniqueName: \"kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587638    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5jp\" (UniqueName: \"kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587690    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4rl\" (UniqueName: \"kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587717    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvhn\" (UniqueName: \"kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587766    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587794    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfpg\" (UniqueName: \"kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587837    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhw8\" (UniqueName: \"kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587872    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587903    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8r4w\" (UniqueName: \"kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.587950    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588023    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.588099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588071    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlcd9\" (UniqueName: \"kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588107    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588133    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxwg\" (UniqueName: \"kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588174    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588241    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqtv\" (UniqueName: \"kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588275    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588323    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588352    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgxn\" (UniqueName: \"kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn\") pod \"maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588379    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmq65\" (UniqueName: \"kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.588801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.588514    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.589286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.589099    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.589286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.589158    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vts8\" (UniqueName: \"kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.590590 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.589979    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.590590 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.590349    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.590758 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.590712    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.590823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.590792    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.590884 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.590842    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"]
Oct 30 04:00:00.591492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.591347    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.610193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.609965    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kxw\" (UniqueName: \"kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.618475 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.614736    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5jp\" (UniqueName: \"kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp\") pod \"maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.629541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.629515    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:00.632127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.631823    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmq65\" (UniqueName: \"kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65\") pod \"maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.640085 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.638059    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4rl\" (UniqueName: \"kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.656505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.654461    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfpg\" (UniqueName: \"kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg\") pod \"maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.662281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.662250    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:00.668344 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.668320    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgxn\" (UniqueName: \"kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn\") pod \"maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.693253 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.693209    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:00.699830 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.699775    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.700087 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.700062    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.701697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.701672    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvhr\" (UniqueName: \"kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr\") pod \"maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.701865 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.701847    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.702747    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.702798    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvhn\" (UniqueName: \"kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.702876    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhw8\" (UniqueName: \"kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.702912    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.702948    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8r4w\" (UniqueName: \"kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703045    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703081    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlcd9\" (UniqueName: \"kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703117    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxwg\" (UniqueName: \"kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703157    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703222    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqtv\" (UniqueName: \"kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703254    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703323    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703358    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vts8\" (UniqueName: \"kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703460    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7r9t\" (UniqueName: \"kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.703556    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.704393    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.705524    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.706205    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.700439    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.706354    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.701155    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.707502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.707402    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.709881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.709745    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.714741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.712072    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.729125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.728438    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhw8\" (UniqueName: \"kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8\") pod \"maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.730261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.729840    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvhn\" (UniqueName: \"kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn\") pod \"maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.733517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.733464    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvhr\" (UniqueName: \"kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr\") pod \"maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.737233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.737115    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxwg\" (UniqueName: \"kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg\") pod \"maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.738860 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.737501    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:00.738860 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.738821    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7r9t\" (UniqueName: \"kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t\") pod \"maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.744572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.744056    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:00.751926 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.751872    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vts8\" (UniqueName: \"kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.755902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.752712    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:00.762465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.761105    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8r4w\" (UniqueName: \"kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w\") pod \"maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.762465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.762017    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlcd9\" (UniqueName: \"kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:00.762465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.762361    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqtv\" (UniqueName: \"kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:00.763105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.762958    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:00.778825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.773148    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:00.782191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.782162    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:00.803247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.802988    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:00.806789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.806567    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:00.887154 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.887126    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:00.898278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.897458    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"]
Oct 30 04:00:00.905415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.903024    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:00.931708 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.931678    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:00.956098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.955935    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:00.996693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:00.995961    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:01.043009 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.042983    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:01.163336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.163306    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"]
Oct 30 04:00:01.218702 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:01.218583    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c021a4_3f0f_4c64_878a_d9e9cd4d4efd.slice/crio-9b93d13f0ff30c09c4dee0332e20d256f11237d79bfad56bd803d03e8d4480d0 WatchSource:0}: Error finding container 9b93d13f0ff30c09c4dee0332e20d256f11237d79bfad56bd803d03e8d4480d0: Status 404 returned error can't find the container with id 9b93d13f0ff30c09c4dee0332e20d256f11237d79bfad56bd803d03e8d4480d0
Oct 30 04:00:01.330753 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.330659    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"]
Oct 30 04:00:01.341306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.340490    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"]
Oct 30 04:00:01.348302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.347989    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"]
Oct 30 04:00:01.646762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.646258    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerStarted","Data":"43fd94e694e859e2d82a55e3b321b0cb16d54353a48de14fcdcb2542d637a6ff"}
Oct 30 04:00:01.648569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.648187    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerStarted","Data":"2256c7f2530292578c866c9172e71c0b14ac94339e8e62f429605646fd6162df"}
Oct 30 04:00:01.650090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.650016    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerStarted","Data":"069195a4094677124596bde68a02d79e552532e11307d67bfe9e203367cc48f2"}
Oct 30 04:00:01.651832 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.651813    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerStarted","Data":"26d5d0d70f08abe4c16fada6d2794615145924f4fded52db84e7e272c90b2c06"}
Oct 30 04:00:01.654710 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.653964    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerStarted","Data":"9b93d13f0ff30c09c4dee0332e20d256f11237d79bfad56bd803d03e8d4480d0"}
Oct 30 04:00:01.668837 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.668763    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"]
Oct 30 04:00:01.682843 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:01.682812    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722060d3_d60d_4c96_972d_71e3b1408128.slice/crio-0a4b9c0f6937a3d373f30ed43d7975bc2b8b6763911a66a3cdcab53df17caef3 WatchSource:0}: Error finding container 0a4b9c0f6937a3d373f30ed43d7975bc2b8b6763911a66a3cdcab53df17caef3: Status 404 returned error can't find the container with id 0a4b9c0f6937a3d373f30ed43d7975bc2b8b6763911a66a3cdcab53df17caef3
Oct 30 04:00:01.684075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.684056    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"]
Oct 30 04:00:01.692348 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:01.692240    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ced66d5_2223_4af9_9a6d_9de05d1f6750.slice/crio-828ce6bbd6735d67312588b4f3cc5e82c90e8a6d78961c2334433b63859b8750 WatchSource:0}: Error finding container 828ce6bbd6735d67312588b4f3cc5e82c90e8a6d78961c2334433b63859b8750: Status 404 returned error can't find the container with id 828ce6bbd6735d67312588b4f3cc5e82c90e8a6d78961c2334433b63859b8750
Oct 30 04:00:01.737441 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.735077    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"]
Oct 30 04:00:01.891538 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.891510    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"]
Oct 30 04:00:01.898490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.898462    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"]
Oct 30 04:00:01.903963 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:01.903917    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"]
Oct 30 04:00:01.997579 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:01.997550    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf0d3e6_b75e_4652_a4f1_a348c357dc9e.slice/crio-5d0da999ea73a7c939fa06e5715892050da31b787e62bc61db094b0d275e4fed WatchSource:0}: Error finding container 5d0da999ea73a7c939fa06e5715892050da31b787e62bc61db094b0d275e4fed: Status 404 returned error can't find the container with id 5d0da999ea73a7c939fa06e5715892050da31b787e62bc61db094b0d275e4fed
Oct 30 04:00:02.057251 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.057226    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"]
Oct 30 04:00:02.065281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.065244    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"]
Oct 30 04:00:02.067054 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:02.067026    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5316ebf_1c98_4b55_af72_243224166688.slice/crio-0c88fc08c65d545baea77ecf506d4ced74abebe040429c821cc4fe9407d8bc00 WatchSource:0}: Error finding container 0c88fc08c65d545baea77ecf506d4ced74abebe040429c821cc4fe9407d8bc00: Status 404 returned error can't find the container with id 0c88fc08c65d545baea77ecf506d4ced74abebe040429c821cc4fe9407d8bc00
Oct 30 04:00:02.071800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.071775    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"]
Oct 30 04:00:02.092915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.092129    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"]
Oct 30 04:00:02.105237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.105211    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"]
Oct 30 04:00:02.110856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.110834    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"]
Oct 30 04:00:02.144766 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.144737    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"]
Oct 30 04:00:02.149697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.149669    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"]
Oct 30 04:00:02.193981 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.193951    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"]
Oct 30 04:00:02.398140 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:02.398061    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836b510f_a531_4569_9bc1_3416e47cafee.slice/crio-81cdd8caa266e4c89ada4d58e6b3c9c6c1fb6c0984041de44e560eadcdce5ef3 WatchSource:0}: Error finding container 81cdd8caa266e4c89ada4d58e6b3c9c6c1fb6c0984041de44e560eadcdce5ef3: Status 404 returned error can't find the container with id 81cdd8caa266e4c89ada4d58e6b3c9c6c1fb6c0984041de44e560eadcdce5ef3
Oct 30 04:00:02.483601 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:02.483559    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e98105c_1514_413e_9bce_2cbc6761294a.slice/crio-ac804718e9794e894cb018d863385fb795e82e844edc1c6ff2b10b932dfa5faa WatchSource:0}: Error finding container ac804718e9794e894cb018d863385fb795e82e844edc1c6ff2b10b932dfa5faa: Status 404 returned error can't find the container with id ac804718e9794e894cb018d863385fb795e82e844edc1c6ff2b10b932dfa5faa
Oct 30 04:00:02.489523 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:02.489496    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c8e20a_d1f8_4c0f_bf6f_fff8aa2c18f9.slice/crio-0f25251c1d4d512e2e282c1b25c29f1986ce246420a465c139a1cd6c96a71632 WatchSource:0}: Error finding container 0f25251c1d4d512e2e282c1b25c29f1986ce246420a465c139a1cd6c96a71632: Status 404 returned error can't find the container with id 0f25251c1d4d512e2e282c1b25c29f1986ce246420a465c139a1cd6c96a71632
Oct 30 04:00:02.494495 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:02.493713    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989c1932_7a7a_4f64_93f5_6a46bd823d89.slice/crio-0676e712015597058620f789e42a1b6fe126d988a4ff76ccd60b78476f1d6fa6 WatchSource:0}: Error finding container 0676e712015597058620f789e42a1b6fe126d988a4ff76ccd60b78476f1d6fa6: Status 404 returned error can't find the container with id 0676e712015597058620f789e42a1b6fe126d988a4ff76ccd60b78476f1d6fa6
Oct 30 04:00:02.498697 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:02.498667    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24f3539_e5a7_4de6_9c42_ca52c8de6344.slice/crio-381aaa31d64b1d0a654799c59db0bbe7718d7f9b629c47e7d4f515524a753556 WatchSource:0}: Error finding container 381aaa31d64b1d0a654799c59db0bbe7718d7f9b629c47e7d4f515524a753556: Status 404 returned error can't find the container with id 381aaa31d64b1d0a654799c59db0bbe7718d7f9b629c47e7d4f515524a753556
Oct 30 04:00:02.690796 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.690759    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerStarted","Data":"b1dc3b28bccaa71c77972ab26f9401b2d029e13af432a190d299fd19681a7a99"}
Oct 30 04:00:02.694196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.694102    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerStarted","Data":"381aaa31d64b1d0a654799c59db0bbe7718d7f9b629c47e7d4f515524a753556"}
Oct 30 04:00:02.699148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.698546    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerStarted","Data":"dc6ca39ac64292c6bed794ea0a09c4fc5d2d53bf0fec83b0107591888bd697e1"}
Oct 30 04:00:02.703093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.703061    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerStarted","Data":"5d0da999ea73a7c939fa06e5715892050da31b787e62bc61db094b0d275e4fed"}
Oct 30 04:00:02.707881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.706891    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerStarted","Data":"9e282ba0edd333f3024e0ec93f511081fdadf6d4ce1792c12f5b9fc3bd753b14"}
Oct 30 04:00:02.710519 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.710433    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerStarted","Data":"31630caa83241fc1e4f575883cd9fba93e4d3436280b2c427614b02e3b85c445"}
Oct 30 04:00:02.720923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.720898    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerStarted","Data":"7d040a96d96869c50d9e7f66029513f107ed4d02bf4beb3d41ed8540c75df461"}
Oct 30 04:00:02.721004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.720945    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerStarted","Data":"0a4b9c0f6937a3d373f30ed43d7975bc2b8b6763911a66a3cdcab53df17caef3"}
Oct 30 04:00:02.725568 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.724661    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerStarted","Data":"8505f919a74895c0236920f27a4866fd65390d76ebeb9c1d5462c6d142d0c9e6"}
Oct 30 04:00:02.725568 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.725045    4877 scope.go:122] "RemoveContainer" containerID="8505f919a74895c0236920f27a4866fd65390d76ebeb9c1d5462c6d142d0c9e6"
Oct 30 04:00:02.728410 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.728356    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" podStartSLOduration=2.7283403550000003 podStartE2EDuration="2.728340355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:02.723864894 +0000 UTC m=+3431.152754323" watchObservedRunningTime="2026-10-30 04:00:02.728340355 +0000 UTC m=+3431.157229774"
Oct 30 04:00:02.731769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.731743    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerStarted","Data":"dcb845296025b93d8298c178922dc92908673f08d90e7eb55a40ef806758e540"}
Oct 30 04:00:02.741650 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.741622    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" event={"ID":"e7409641-adac-4021-97f5-b3fcffb172a8","Type":"ContainerStarted","Data":"bd06cf219d1638d43175e8d5e614d27b027f7bc5612668e0028dc46dd7e725ab"}
Oct 30 04:00:02.743507 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.743478    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerStarted","Data":"0676e712015597058620f789e42a1b6fe126d988a4ff76ccd60b78476f1d6fa6"}
Oct 30 04:00:02.745404 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.745380    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerStarted","Data":"81cdd8caa266e4c89ada4d58e6b3c9c6c1fb6c0984041de44e560eadcdce5ef3"}
Oct 30 04:00:02.747074 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.747050    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerStarted","Data":"a15d52cdb71ef31baa93fb220d61e9196d78b645ee151e4ecf6ab2dcc6233833"}
Oct 30 04:00:02.749219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.749193    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerStarted","Data":"fb58fe75dc0304ee926990a6a8f8d5cea28210de92b9df63f6548876d1919628"}
Oct 30 04:00:02.751788 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.751764    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerStarted","Data":"ac804718e9794e894cb018d863385fb795e82e844edc1c6ff2b10b932dfa5faa"}
Oct 30 04:00:02.753592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.753564    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerStarted","Data":"0f25251c1d4d512e2e282c1b25c29f1986ce246420a465c139a1cd6c96a71632"}
Oct 30 04:00:02.755159 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.755127    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" event={"ID":"008f7da2-1318-40d9-b589-bb94835510e6","Type":"ContainerStarted","Data":"6ac4c1136d64ffcb5b2e0941f313f5f98dca2b147e4c0474b8b45b05503b25b5"}
Oct 30 04:00:02.757372 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.757347    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" event={"ID":"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c","Type":"ContainerStarted","Data":"f8545c4f984954bf614b2cd1d3a334a8422c5da90b2a97d4ae026c86d901eca1"}
Oct 30 04:00:02.761104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.761078    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" event={"ID":"e5316ebf-1c98-4b55-af72-243224166688","Type":"ContainerStarted","Data":"0c88fc08c65d545baea77ecf506d4ced74abebe040429c821cc4fe9407d8bc00"}
Oct 30 04:00:02.763216 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.763194    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerStarted","Data":"8f72eb5613306c986a40bfab8a14a5a437b3ac53e73b8f1dbd36bba509798c40"}
Oct 30 04:00:02.763295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.763235    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerStarted","Data":"828ce6bbd6735d67312588b4f3cc5e82c90e8a6d78961c2334433b63859b8750"}
Oct 30 04:00:02.771697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.771645    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" podStartSLOduration=2.771631438 podStartE2EDuration="2.771631438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:02.756824345 +0000 UTC m=+3431.185713762" watchObservedRunningTime="2026-10-30 04:00:02.771631438 +0000 UTC m=+3431.200520856"
Oct 30 04:00:02.815849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.815615    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" podStartSLOduration=2.815536019 podStartE2EDuration="2.815536019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:02.807878339 +0000 UTC m=+3431.236767750" watchObservedRunningTime="2026-10-30 04:00:02.815536019 +0000 UTC m=+3431.244425435"
Oct 30 04:00:02.850416 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.850286    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" podStartSLOduration=2.850265356 podStartE2EDuration="2.850265356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:02.847083557 +0000 UTC m=+3431.275972978" watchObservedRunningTime="2026-10-30 04:00:02.850265356 +0000 UTC m=+3431.279154775"
Oct 30 04:00:02.933038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.932966    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" podStartSLOduration=2.93294121 podStartE2EDuration="2.93294121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:02.928677318 +0000 UTC m=+3431.357566733" watchObservedRunningTime="2026-10-30 04:00:02.93294121 +0000 UTC m=+3431.361830633"
Oct 30 04:00:02.962208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:02.962147    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" podStartSLOduration=2.962130041 podStartE2EDuration="2.962130041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:02.96191808 +0000 UTC m=+3431.390807497" watchObservedRunningTime="2026-10-30 04:00:02.962130041 +0000 UTC m=+3431.391019463"
Oct 30 04:00:03.792443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.788769    4877 generic.go:356] "Generic (PLEG): container finished" podID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerID="9dc6429d05efef39f05a15f064f9ba44752241ec1e8967c666af259f8296580d" exitCode=60
Oct 30 04:00:03.792443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.788851    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" event={"ID":"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c","Type":"ContainerDied","Data":"9dc6429d05efef39f05a15f064f9ba44752241ec1e8967c666af259f8296580d"}
Oct 30 04:00:03.792443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.789247    4877 scope.go:122] "RemoveContainer" containerID="9dc6429d05efef39f05a15f064f9ba44752241ec1e8967c666af259f8296580d"
Oct 30 04:00:03.794911 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.794049    4877 generic.go:356] "Generic (PLEG): container finished" podID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerID="8f72eb5613306c986a40bfab8a14a5a437b3ac53e73b8f1dbd36bba509798c40" exitCode=60
Oct 30 04:00:03.794911 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.794172    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerDied","Data":"8f72eb5613306c986a40bfab8a14a5a437b3ac53e73b8f1dbd36bba509798c40"}
Oct 30 04:00:03.794911 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.794523    4877 scope.go:122] "RemoveContainer" containerID="8f72eb5613306c986a40bfab8a14a5a437b3ac53e73b8f1dbd36bba509798c40"
Oct 30 04:00:03.800211 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.800184    4877 generic.go:356] "Generic (PLEG): container finished" podID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerID="b1dc3b28bccaa71c77972ab26f9401b2d029e13af432a190d299fd19681a7a99" exitCode=60
Oct 30 04:00:03.800283 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.800266    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerDied","Data":"b1dc3b28bccaa71c77972ab26f9401b2d029e13af432a190d299fd19681a7a99"}
Oct 30 04:00:03.800696 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.800677    4877 scope.go:122] "RemoveContainer" containerID="b1dc3b28bccaa71c77972ab26f9401b2d029e13af432a190d299fd19681a7a99"
Oct 30 04:00:03.805710 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.805684    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerStarted","Data":"4bde488a0d3f6056ce8b1ca0d6478d492e3d4f0dbdd93768357de150d1752b15"}
Oct 30 04:00:03.806068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.806049    4877 scope.go:122] "RemoveContainer" containerID="4bde488a0d3f6056ce8b1ca0d6478d492e3d4f0dbdd93768357de150d1752b15"
Oct 30 04:00:03.808577 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.808555    4877 generic.go:356] "Generic (PLEG): container finished" podID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerID="de15eebd3bc3f5e357a21a475fdc03f7a3dca6fa12d35c73065747aef5298a7e" exitCode=60
Oct 30 04:00:03.808866 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.808807    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerDied","Data":"de15eebd3bc3f5e357a21a475fdc03f7a3dca6fa12d35c73065747aef5298a7e"}
Oct 30 04:00:03.808956 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.808883    4877 scope.go:122] "RemoveContainer" containerID="de15eebd3bc3f5e357a21a475fdc03f7a3dca6fa12d35c73065747aef5298a7e"
Oct 30 04:00:03.810992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.810969    4877 generic.go:356] "Generic (PLEG): container finished" podID="fe99e906-f412-4f34-91d1-c00607c1f067" containerID="9e282ba0edd333f3024e0ec93f511081fdadf6d4ce1792c12f5b9fc3bd753b14" exitCode=60
Oct 30 04:00:03.811095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.811066    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerDied","Data":"9e282ba0edd333f3024e0ec93f511081fdadf6d4ce1792c12f5b9fc3bd753b14"}
Oct 30 04:00:03.811365 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.811344    4877 scope.go:122] "RemoveContainer" containerID="9e282ba0edd333f3024e0ec93f511081fdadf6d4ce1792c12f5b9fc3bd753b14"
Oct 30 04:00:03.813958 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.813935    4877 generic.go:356] "Generic (PLEG): container finished" podID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerID="31630caa83241fc1e4f575883cd9fba93e4d3436280b2c427614b02e3b85c445" exitCode=60
Oct 30 04:00:03.814047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.814016    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerDied","Data":"31630caa83241fc1e4f575883cd9fba93e4d3436280b2c427614b02e3b85c445"}
Oct 30 04:00:03.814348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.814330    4877 scope.go:122] "RemoveContainer" containerID="31630caa83241fc1e4f575883cd9fba93e4d3436280b2c427614b02e3b85c445"
Oct 30 04:00:03.819051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.819026    4877 generic.go:356] "Generic (PLEG): container finished" podID="722060d3-d60d-4c96-972d-71e3b1408128" containerID="7d040a96d96869c50d9e7f66029513f107ed4d02bf4beb3d41ed8540c75df461" exitCode=60
Oct 30 04:00:03.819147 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.819130    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerDied","Data":"7d040a96d96869c50d9e7f66029513f107ed4d02bf4beb3d41ed8540c75df461"}
Oct 30 04:00:03.819489 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.819472    4877 scope.go:122] "RemoveContainer" containerID="7d040a96d96869c50d9e7f66029513f107ed4d02bf4beb3d41ed8540c75df461"
Oct 30 04:00:03.821836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.821813    4877 generic.go:356] "Generic (PLEG): container finished" podID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerID="8505f919a74895c0236920f27a4866fd65390d76ebeb9c1d5462c6d142d0c9e6" exitCode=60
Oct 30 04:00:03.822017 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.821998    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerDied","Data":"8505f919a74895c0236920f27a4866fd65390d76ebeb9c1d5462c6d142d0c9e6"}
Oct 30 04:00:03.836893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.836708    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerID="b59f4a88428028c7e8327ee4dbf34103c211668070004ce61b357e55adba7735" exitCode=60
Oct 30 04:00:03.836983 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.836908    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerDied","Data":"b59f4a88428028c7e8327ee4dbf34103c211668070004ce61b357e55adba7735"}
Oct 30 04:00:03.839214 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.839038    4877 generic.go:356] "Generic (PLEG): container finished" podID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerID="fb58fe75dc0304ee926990a6a8f8d5cea28210de92b9df63f6548876d1919628" exitCode=60
Oct 30 04:00:03.839214 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.839094    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerDied","Data":"fb58fe75dc0304ee926990a6a8f8d5cea28210de92b9df63f6548876d1919628"}
Oct 30 04:00:03.892554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.882914    4877 scope.go:122] "RemoveContainer" containerID="b59f4a88428028c7e8327ee4dbf34103c211668070004ce61b357e55adba7735"
Oct 30 04:00:03.892554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:03.883668    4877 scope.go:122] "RemoveContainer" containerID="fb58fe75dc0304ee926990a6a8f8d5cea28210de92b9df63f6548876d1919628"
Oct 30 04:00:04.860271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.860233    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerStarted","Data":"954669dcdd37976e0aa4d1439d906530cacb93cfb03b08dc62377284a5334777"}
Oct 30 04:00:04.862168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.862137    4877 generic.go:356] "Generic (PLEG): container finished" podID="008f7da2-1318-40d9-b589-bb94835510e6" containerID="aab2d9c0e724c2f74ebc7aa39a7e0f1b04c3cd13c38f5c2042fbda092e2a5f36" exitCode=60
Oct 30 04:00:04.862293 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.862216    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" event={"ID":"008f7da2-1318-40d9-b589-bb94835510e6","Type":"ContainerDied","Data":"aab2d9c0e724c2f74ebc7aa39a7e0f1b04c3cd13c38f5c2042fbda092e2a5f36"}
Oct 30 04:00:04.862543 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.862524    4877 scope.go:122] "RemoveContainer" containerID="aab2d9c0e724c2f74ebc7aa39a7e0f1b04c3cd13c38f5c2042fbda092e2a5f36"
Oct 30 04:00:04.864655 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.864626    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" event={"ID":"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c","Type":"ContainerStarted","Data":"20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb"}
Oct 30 04:00:04.866374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.866348    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" event={"ID":"e5316ebf-1c98-4b55-af72-243224166688","Type":"ContainerStarted","Data":"75af4034906a966f36b6564df6d336de1bcea1561968c00da63c0a2e6e7b9f47"}
Oct 30 04:00:04.868419 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.868398    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerStarted","Data":"e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2"}
Oct 30 04:00:04.870369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.870350    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerStarted","Data":"a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323"}
Oct 30 04:00:04.872035 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.872012    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerStarted","Data":"7419fe0f79001312aa562a10f8fc2bac288e410fc5ec752d4f832516cfe49c06"}
Oct 30 04:00:04.874037 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.874013    4877 generic.go:356] "Generic (PLEG): container finished" podID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerID="4bde488a0d3f6056ce8b1ca0d6478d492e3d4f0dbdd93768357de150d1752b15" exitCode=60
Oct 30 04:00:04.874136 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.874104    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerDied","Data":"4bde488a0d3f6056ce8b1ca0d6478d492e3d4f0dbdd93768357de150d1752b15"}
Oct 30 04:00:04.874197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.874148    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerStarted","Data":"86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329"}
Oct 30 04:00:04.876444 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.876398    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerStarted","Data":"a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df"}
Oct 30 04:00:04.903613 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.903556    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" podStartSLOduration=4.9035416 podStartE2EDuration="4.9035416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:04.896762923 +0000 UTC m=+3433.325652341" watchObservedRunningTime="2026-10-30 04:00:04.9035416 +0000 UTC m=+3433.332431049"
Oct 30 04:00:04.905741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.905711    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerStarted","Data":"e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a"}
Oct 30 04:00:04.908271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.908246    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerStarted","Data":"3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4"}
Oct 30 04:00:04.911072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.911040    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerStarted","Data":"71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6"}
Oct 30 04:00:04.913569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.913543    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerStarted","Data":"edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0"}
Oct 30 04:00:04.915573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.915548    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerStarted","Data":"55a85e7dc354128001229aa80125d3bb1a5c381b786937d24cddd794a18d7f4d"}
Oct 30 04:00:04.917642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.917616    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7409641-adac-4021-97f5-b3fcffb172a8" containerID="ee1fbd2503d33b6fe95543d7d322966063203508f99c5ffd117a12fb70dac3b5" exitCode=60
Oct 30 04:00:04.917733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.917720    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" event={"ID":"e7409641-adac-4021-97f5-b3fcffb172a8","Type":"ContainerDied","Data":"ee1fbd2503d33b6fe95543d7d322966063203508f99c5ffd117a12fb70dac3b5"}
Oct 30 04:00:04.917971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.917948    4877 scope.go:122] "RemoveContainer" containerID="ee1fbd2503d33b6fe95543d7d322966063203508f99c5ffd117a12fb70dac3b5"
Oct 30 04:00:04.919898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.919878    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerStarted","Data":"d5dcb448c0a7bc09e60f08892561ce932ea522d18b37e5b62fb40ab9837d1a6f"}
Oct 30 04:00:04.922069 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.922036    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerStarted","Data":"018b29ecc6b24aca38c06e5f032c4bab95918d718a2ed3af713afc7499348a3c"}
Oct 30 04:00:04.924885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.924859    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerStarted","Data":"4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f"}
Oct 30 04:00:04.928030 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.928007    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerStarted","Data":"74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd"}
Oct 30 04:00:04.930901 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.930879    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerStarted","Data":"b56b9613e98505f6920f3bfc3e9a322cadb068fdeb069eda6cce4fd3f04902b5"}
Oct 30 04:00:04.966302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:04.966245    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" podStartSLOduration=4.966230373 podStartE2EDuration="4.966230373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:04.962168789 +0000 UTC m=+3433.391058207" watchObservedRunningTime="2026-10-30 04:00:04.966230373 +0000 UTC m=+3433.395119882"
Oct 30 04:00:05.065959 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.065847    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" podStartSLOduration=5.065826761 podStartE2EDuration="5.065826761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.010805454 +0000 UTC m=+3433.439694873" watchObservedRunningTime="2026-10-30 04:00:05.065826761 +0000 UTC m=+3433.494716179"
Oct 30 04:00:05.069471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.069402    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" podStartSLOduration=5.069391393 podStartE2EDuration="5.069391393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.05433941 +0000 UTC m=+3433.483228830" watchObservedRunningTime="2026-10-30 04:00:05.069391393 +0000 UTC m=+3433.498280811"
Oct 30 04:00:05.107679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.107619    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" podStartSLOduration=5.107599676 podStartE2EDuration="5.107599676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.105660908 +0000 UTC m=+3433.534550326" watchObservedRunningTime="2026-10-30 04:00:05.107599676 +0000 UTC m=+3433.536489095"
Oct 30 04:00:05.200306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.196401    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" podStartSLOduration=5.196378658 podStartE2EDuration="5.196378658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.174569443 +0000 UTC m=+3433.603458887" watchObservedRunningTime="2026-10-30 04:00:05.196378658 +0000 UTC m=+3433.625268078"
Oct 30 04:00:05.256617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.256565    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" podStartSLOduration=5.256552061 podStartE2EDuration="5.256552061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.255157902 +0000 UTC m=+3433.684047320" watchObservedRunningTime="2026-10-30 04:00:05.256552061 +0000 UTC m=+3433.685441479"
Oct 30 04:00:05.372361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.372306    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" podStartSLOduration=5.372292231 podStartE2EDuration="5.372292231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.334738413 +0000 UTC m=+3433.763627831" watchObservedRunningTime="2026-10-30 04:00:05.372292231 +0000 UTC m=+3433.801181648"
Oct 30 04:00:05.408017 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.404964    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" podStartSLOduration=5.404940315 podStartE2EDuration="5.404940315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.400542281 +0000 UTC m=+3433.829431724" watchObservedRunningTime="2026-10-30 04:00:05.404940315 +0000 UTC m=+3433.833829734"
Oct 30 04:00:05.469940 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.469890    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" podStartSLOduration=5.46987519 podStartE2EDuration="5.46987519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.463893207 +0000 UTC m=+3433.892782625" watchObservedRunningTime="2026-10-30 04:00:05.46987519 +0000 UTC m=+3433.898764608"
Oct 30 04:00:05.569612 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.569556    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" podStartSLOduration=5.569543245 podStartE2EDuration="5.569543245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.566494717 +0000 UTC m=+3433.995384137" watchObservedRunningTime="2026-10-30 04:00:05.569543245 +0000 UTC m=+3433.998432663"
Oct 30 04:00:05.601046 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.600938    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" podStartSLOduration=5.6009247 podStartE2EDuration="5.6009247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:05.599395454 +0000 UTC m=+3434.028284872" watchObservedRunningTime="2026-10-30 04:00:05.6009247 +0000 UTC m=+3434.029814167"
Oct 30 04:00:05.946292 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.946259    4877 generic.go:356] "Generic (PLEG): container finished" podID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerID="edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0" exitCode=60
Oct 30 04:00:05.946849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.946335    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerDied","Data":"edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0"}
Oct 30 04:00:05.946849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.946398    4877 scope.go:122] "RemoveContainer" containerID="8505f919a74895c0236920f27a4866fd65390d76ebeb9c1d5462c6d142d0c9e6"
Oct 30 04:00:05.946849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.946709    4877 scope.go:122] "RemoveContainer" containerID="edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0"
Oct 30 04:00:05.947163 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:05.947131    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt_odh-ai-gateway-infra(f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd"
Oct 30 04:00:05.948340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.948306    4877 generic.go:356] "Generic (PLEG): container finished" podID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerID="55a85e7dc354128001229aa80125d3bb1a5c381b786937d24cddd794a18d7f4d" exitCode=60
Oct 30 04:00:05.948461 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.948378    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerDied","Data":"55a85e7dc354128001229aa80125d3bb1a5c381b786937d24cddd794a18d7f4d"}
Oct 30 04:00:05.948690 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.948675    4877 scope.go:122] "RemoveContainer" containerID="55a85e7dc354128001229aa80125d3bb1a5c381b786937d24cddd794a18d7f4d"
Oct 30 04:00:05.950142 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.950120    4877 generic.go:356] "Generic (PLEG): container finished" podID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerID="d5dcb448c0a7bc09e60f08892561ce932ea522d18b37e5b62fb40ab9837d1a6f" exitCode=60
Oct 30 04:00:05.950231 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.950201    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerDied","Data":"d5dcb448c0a7bc09e60f08892561ce932ea522d18b37e5b62fb40ab9837d1a6f"}
Oct 30 04:00:05.950480 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.950452    4877 scope.go:122] "RemoveContainer" containerID="d5dcb448c0a7bc09e60f08892561ce932ea522d18b37e5b62fb40ab9837d1a6f"
Oct 30 04:00:05.952455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.952401    4877 generic.go:356] "Generic (PLEG): container finished" podID="836b510f-a531-4569-9bc1-3416e47cafee" containerID="018b29ecc6b24aca38c06e5f032c4bab95918d718a2ed3af713afc7499348a3c" exitCode=60
Oct 30 04:00:05.952535 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.952484    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerDied","Data":"018b29ecc6b24aca38c06e5f032c4bab95918d718a2ed3af713afc7499348a3c"}
Oct 30 04:00:05.952741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.952726    4877 scope.go:122] "RemoveContainer" containerID="018b29ecc6b24aca38c06e5f032c4bab95918d718a2ed3af713afc7499348a3c"
Oct 30 04:00:05.954775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.954750    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerID="4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f" exitCode=60
Oct 30 04:00:05.954885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.954777    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerDied","Data":"4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f"}
Oct 30 04:00:05.955068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.955049    4877 scope.go:122] "RemoveContainer" containerID="4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f"
Oct 30 04:00:05.955306 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:05.955285    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6_odh-ai-gateway-infra(e7a919ba-751f-4778-8d8c-7cd445d1c3d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2"
Oct 30 04:00:05.956986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.956957    4877 generic.go:356] "Generic (PLEG): container finished" podID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerID="74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd" exitCode=60
Oct 30 04:00:05.957101 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.956990    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerDied","Data":"74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd"}
Oct 30 04:00:05.957251 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.957233    4877 scope.go:122] "RemoveContainer" containerID="74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd"
Oct 30 04:00:05.957510 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:05.957490    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc_odh-ai-gateway-infra(e81dceac-bb98-4f7c-9995-d91a2fef84d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0"
Oct 30 04:00:05.959388 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.959365    4877 generic.go:356] "Generic (PLEG): container finished" podID="1e98105c-1514-413e-9bce-2cbc6761294a" containerID="b56b9613e98505f6920f3bfc3e9a322cadb068fdeb069eda6cce4fd3f04902b5" exitCode=60
Oct 30 04:00:05.959548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.959526    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerDied","Data":"b56b9613e98505f6920f3bfc3e9a322cadb068fdeb069eda6cce4fd3f04902b5"}
Oct 30 04:00:05.959834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.959813    4877 scope.go:122] "RemoveContainer" containerID="b56b9613e98505f6920f3bfc3e9a322cadb068fdeb069eda6cce4fd3f04902b5"
Oct 30 04:00:05.961999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.961978    4877 generic.go:356] "Generic (PLEG): container finished" podID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerID="954669dcdd37976e0aa4d1439d906530cacb93cfb03b08dc62377284a5334777" exitCode=60
Oct 30 04:00:05.962082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.962055    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerDied","Data":"954669dcdd37976e0aa4d1439d906530cacb93cfb03b08dc62377284a5334777"}
Oct 30 04:00:05.962288 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.962272    4877 scope.go:122] "RemoveContainer" containerID="954669dcdd37976e0aa4d1439d906530cacb93cfb03b08dc62377284a5334777"
Oct 30 04:00:05.964393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.964375    4877 generic.go:356] "Generic (PLEG): container finished" podID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerID="20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb" exitCode=60
Oct 30 04:00:05.964487 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.964465    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" event={"ID":"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c","Type":"ContainerDied","Data":"20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb"}
Oct 30 04:00:05.964778 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.964759    4877 scope.go:122] "RemoveContainer" containerID="20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb"
Oct 30 04:00:05.965045 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:05.965023    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8_odh-ai-gateway-infra(5331a17c-1fa3-46ae-bba8-ec3d5f94f68c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c"
Oct 30 04:00:05.966266 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.966246    4877 generic.go:356] "Generic (PLEG): container finished" podID="e5316ebf-1c98-4b55-af72-243224166688" containerID="75af4034906a966f36b6564df6d336de1bcea1561968c00da63c0a2e6e7b9f47" exitCode=60
Oct 30 04:00:05.966365 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.966329    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" event={"ID":"e5316ebf-1c98-4b55-af72-243224166688","Type":"ContainerDied","Data":"75af4034906a966f36b6564df6d336de1bcea1561968c00da63c0a2e6e7b9f47"}
Oct 30 04:00:05.966520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.966504    4877 scope.go:122] "RemoveContainer" containerID="75af4034906a966f36b6564df6d336de1bcea1561968c00da63c0a2e6e7b9f47"
Oct 30 04:00:05.968778 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.968755    4877 generic.go:356] "Generic (PLEG): container finished" podID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerID="e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2" exitCode=60
Oct 30 04:00:05.969055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.969007    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerDied","Data":"e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2"}
Oct 30 04:00:05.969147 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.969130    4877 scope.go:122] "RemoveContainer" containerID="e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2"
Oct 30 04:00:05.969351 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:05.969330    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms_odh-ai-gateway-infra(7ced66d5-2223-4af9-9a6d-9de05d1f6750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750"
Oct 30 04:00:05.987276 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.987247    4877 generic.go:356] "Generic (PLEG): container finished" podID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerID="a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323" exitCode=60
Oct 30 04:00:05.987360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.987331    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerDied","Data":"a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323"}
Oct 30 04:00:05.988017 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:05.987776    4877 scope.go:122] "RemoveContainer" containerID="a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323"
Oct 30 04:00:05.988142 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:05.988117    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n_odh-ai-gateway-infra(d9413205-622e-4cee-82f9-e6c9fd2041fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe"
Oct 30 04:00:06.000740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.000719    4877 scope.go:122] "RemoveContainer" containerID="b59f4a88428028c7e8327ee4dbf34103c211668070004ce61b357e55adba7735"
Oct 30 04:00:06.002291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.002264    4877 generic.go:356] "Generic (PLEG): container finished" podID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerID="7419fe0f79001312aa562a10f8fc2bac288e410fc5ec752d4f832516cfe49c06" exitCode=60
Oct 30 04:00:06.002397 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.002344    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerDied","Data":"7419fe0f79001312aa562a10f8fc2bac288e410fc5ec752d4f832516cfe49c06"}
Oct 30 04:00:06.002857 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.002839    4877 scope.go:122] "RemoveContainer" containerID="7419fe0f79001312aa562a10f8fc2bac288e410fc5ec752d4f832516cfe49c06"
Oct 30 04:00:06.011606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.011580    4877 generic.go:356] "Generic (PLEG): container finished" podID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerID="86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329" exitCode=60
Oct 30 04:00:06.011699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.011615    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerDied","Data":"86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329"}
Oct 30 04:00:06.012088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.012067    4877 scope.go:122] "RemoveContainer" containerID="86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329"
Oct 30 04:00:06.012492 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:06.012466    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86_odh-ai-gateway-infra(aa661a42-aaa6-4ff8-947f-3962a969dac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8"
Oct 30 04:00:06.014315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.014295    4877 generic.go:356] "Generic (PLEG): container finished" podID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerID="a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df" exitCode=60
Oct 30 04:00:06.014607 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.014517    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerDied","Data":"a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df"}
Oct 30 04:00:06.016279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.015770    4877 scope.go:122] "RemoveContainer" containerID="a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df"
Oct 30 04:00:06.016279 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:06.016096    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd_odh-ai-gateway-infra(eaf0d3e6-b75e-4652-a4f1-a348c357dc9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e"
Oct 30 04:00:06.020391 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.020369    4877 generic.go:356] "Generic (PLEG): container finished" podID="fe99e906-f412-4f34-91d1-c00607c1f067" containerID="e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a" exitCode=60
Oct 30 04:00:06.022270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.020842    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerDied","Data":"e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a"}
Oct 30 04:00:06.022270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.021239    4877 scope.go:122] "RemoveContainer" containerID="e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a"
Oct 30 04:00:06.022270 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:06.021510    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c_odh-ai-gateway-infra(fe99e906-f412-4f34-91d1-c00607c1f067)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" podUID="fe99e906-f412-4f34-91d1-c00607c1f067"
Oct 30 04:00:06.023741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.023720    4877 generic.go:356] "Generic (PLEG): container finished" podID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerID="3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4" exitCode=60
Oct 30 04:00:06.023846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.023802    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerDied","Data":"3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4"}
Oct 30 04:00:06.024130 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.024112    4877 scope.go:122] "RemoveContainer" containerID="3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4"
Oct 30 04:00:06.024373 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:06.024351    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8_odh-ai-gateway-infra(ad2f1fa7-7466-4640-935e-f78f3aa91ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5"
Oct 30 04:00:06.027149 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.027127    4877 generic.go:356] "Generic (PLEG): container finished" podID="722060d3-d60d-4c96-972d-71e3b1408128" containerID="71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6" exitCode=60
Oct 30 04:00:06.027239 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.027176    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerDied","Data":"71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6"}
Oct 30 04:00:06.027495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.027476    4877 scope.go:122] "RemoveContainer" containerID="71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6"
Oct 30 04:00:06.027723 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:06.027704    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh_odh-ai-gateway-infra(722060d3-d60d-4c96-972d-71e3b1408128)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" podUID="722060d3-d60d-4c96-972d-71e3b1408128"
Oct 30 04:00:06.095299 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.095274    4877 scope.go:122] "RemoveContainer" containerID="fb58fe75dc0304ee926990a6a8f8d5cea28210de92b9df63f6548876d1919628"
Oct 30 04:00:06.120319 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.119590    4877 scope.go:122] "RemoveContainer" containerID="9dc6429d05efef39f05a15f064f9ba44752241ec1e8967c666af259f8296580d"
Oct 30 04:00:06.141773 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.141749    4877 scope.go:122] "RemoveContainer" containerID="8f72eb5613306c986a40bfab8a14a5a437b3ac53e73b8f1dbd36bba509798c40"
Oct 30 04:00:06.157778 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.157758    4877 scope.go:122] "RemoveContainer" containerID="b1dc3b28bccaa71c77972ab26f9401b2d029e13af432a190d299fd19681a7a99"
Oct 30 04:00:06.174088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.174068    4877 scope.go:122] "RemoveContainer" containerID="4bde488a0d3f6056ce8b1ca0d6478d492e3d4f0dbdd93768357de150d1752b15"
Oct 30 04:00:06.204569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.204540    4877 scope.go:122] "RemoveContainer" containerID="de15eebd3bc3f5e357a21a475fdc03f7a3dca6fa12d35c73065747aef5298a7e"
Oct 30 04:00:06.223742 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.223715    4877 scope.go:122] "RemoveContainer" containerID="9e282ba0edd333f3024e0ec93f511081fdadf6d4ce1792c12f5b9fc3bd753b14"
Oct 30 04:00:06.239226 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.239198    4877 scope.go:122] "RemoveContainer" containerID="31630caa83241fc1e4f575883cd9fba93e4d3436280b2c427614b02e3b85c445"
Oct 30 04:00:06.255857 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:06.255831    4877 scope.go:122] "RemoveContainer" containerID="7d040a96d96869c50d9e7f66029513f107ed4d02bf4beb3d41ed8540c75df461"
Oct 30 04:00:07.066482 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.061845    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerStarted","Data":"4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e"}
Oct 30 04:00:07.066482 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.066109    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerStarted","Data":"35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6"}
Oct 30 04:00:07.068331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.068308    4877 scope.go:122] "RemoveContainer" containerID="4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f"
Oct 30 04:00:07.068648 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.068625    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6_odh-ai-gateway-infra(e7a919ba-751f-4778-8d8c-7cd445d1c3d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2"
Oct 30 04:00:07.071834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.071811    4877 scope.go:122] "RemoveContainer" containerID="74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd"
Oct 30 04:00:07.072113 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.072088    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc_odh-ai-gateway-infra(e81dceac-bb98-4f7c-9995-d91a2fef84d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0"
Oct 30 04:00:07.073542 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.073517    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerStarted","Data":"081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f"}
Oct 30 04:00:07.076316 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.076292    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerStarted","Data":"406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75"}
Oct 30 04:00:07.081347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.080932    4877 generic.go:356] "Generic (PLEG): container finished" podID="008f7da2-1318-40d9-b589-bb94835510e6" containerID="74ffa7c911b878404dbb238a3dbba57540ae4d023c6cd445827adf189e2be615" exitCode=60
Oct 30 04:00:07.081347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.081061    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" event={"ID":"008f7da2-1318-40d9-b589-bb94835510e6","Type":"ContainerDied","Data":"74ffa7c911b878404dbb238a3dbba57540ae4d023c6cd445827adf189e2be615"}
Oct 30 04:00:07.081347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.081105    4877 scope.go:122] "RemoveContainer" containerID="aab2d9c0e724c2f74ebc7aa39a7e0f1b04c3cd13c38f5c2042fbda092e2a5f36"
Oct 30 04:00:07.081624 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.081541    4877 scope.go:122] "RemoveContainer" containerID="74ffa7c911b878404dbb238a3dbba57540ae4d023c6cd445827adf189e2be615"
Oct 30 04:00:07.081796 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.081773    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp_odh-ai-gateway-infra(008f7da2-1318-40d9-b589-bb94835510e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" podUID="008f7da2-1318-40d9-b589-bb94835510e6"
Oct 30 04:00:07.084642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.084618    4877 scope.go:122] "RemoveContainer" containerID="20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb"
Oct 30 04:00:07.084926 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.084902    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8_odh-ai-gateway-infra(5331a17c-1fa3-46ae-bba8-ec3d5f94f68c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c"
Oct 30 04:00:07.087584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.087558    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" event={"ID":"e5316ebf-1c98-4b55-af72-243224166688","Type":"ContainerStarted","Data":"59c138d158ab9a6f852cc664965f4ea281e5fd3dd552ff119e97de6bd4446493"}
Oct 30 04:00:07.090173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.090142    4877 scope.go:122] "RemoveContainer" containerID="e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2"
Oct 30 04:00:07.090378 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.090359    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms_odh-ai-gateway-infra(7ced66d5-2223-4af9-9a6d-9de05d1f6750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750"
Oct 30 04:00:07.252909 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.237825    4877 scope.go:122] "RemoveContainer" containerID="a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323"
Oct 30 04:00:07.252909 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.238321    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n_odh-ai-gateway-infra(d9413205-622e-4cee-82f9-e6c9fd2041fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe"
Oct 30 04:00:07.259886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.259847    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerStarted","Data":"2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb"}
Oct 30 04:00:07.265062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.263601    4877 scope.go:122] "RemoveContainer" containerID="86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329"
Oct 30 04:00:07.265062 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.264038    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86_odh-ai-gateway-infra(aa661a42-aaa6-4ff8-947f-3962a969dac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8"
Oct 30 04:00:07.285778 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.281538    4877 scope.go:122] "RemoveContainer" containerID="a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df"
Oct 30 04:00:07.285778 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.282037    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd_odh-ai-gateway-infra(eaf0d3e6-b75e-4652-a4f1-a348c357dc9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e"
Oct 30 04:00:07.312596 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.312561    4877 scope.go:122] "RemoveContainer" containerID="e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a"
Oct 30 04:00:07.313016 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.312983    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c_odh-ai-gateway-infra(fe99e906-f412-4f34-91d1-c00607c1f067)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" podUID="fe99e906-f412-4f34-91d1-c00607c1f067"
Oct 30 04:00:07.319395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.319346    4877 scope.go:122] "RemoveContainer" containerID="3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4"
Oct 30 04:00:07.320769 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.320507    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8_odh-ai-gateway-infra(ad2f1fa7-7466-4640-935e-f78f3aa91ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5"
Oct 30 04:00:07.329328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.328702    4877 scope.go:122] "RemoveContainer" containerID="71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6"
Oct 30 04:00:07.329328 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.329246    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh_odh-ai-gateway-infra(722060d3-d60d-4c96-972d-71e3b1408128)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" podUID="722060d3-d60d-4c96-972d-71e3b1408128"
Oct 30 04:00:07.335357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.334869    4877 scope.go:122] "RemoveContainer" containerID="edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0"
Oct 30 04:00:07.335357 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.335306    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt_odh-ai-gateway-infra(f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd"
Oct 30 04:00:07.338822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.338786    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerStarted","Data":"1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7"}
Oct 30 04:00:07.343166 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.342365    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7409641-adac-4021-97f5-b3fcffb172a8" containerID="83e5a88615296245a4876560a83b731f7638729ff072ffcd64cf950e62e58640" exitCode=60
Oct 30 04:00:07.343166 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.342411    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" event={"ID":"e7409641-adac-4021-97f5-b3fcffb172a8","Type":"ContainerDied","Data":"83e5a88615296245a4876560a83b731f7638729ff072ffcd64cf950e62e58640"}
Oct 30 04:00:07.343166 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.342502    4877 scope.go:122] "RemoveContainer" containerID="ee1fbd2503d33b6fe95543d7d322966063203508f99c5ffd117a12fb70dac3b5"
Oct 30 04:00:07.343166 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:07.342796    4877 scope.go:122] "RemoveContainer" containerID="83e5a88615296245a4876560a83b731f7638729ff072ffcd64cf950e62e58640"
Oct 30 04:00:07.343166 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:07.343124    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr_odh-ai-gateway-infra(e7409641-adac-4021-97f5-b3fcffb172a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" podUID="e7409641-adac-4021-97f5-b3fcffb172a8"
Oct 30 04:00:08.414452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.404877    4877 generic.go:356] "Generic (PLEG): container finished" podID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerID="2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb" exitCode=60
Oct 30 04:00:08.414452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.405075    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerDied","Data":"2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb"}
Oct 30 04:00:08.414452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.405169    4877 scope.go:122] "RemoveContainer" containerID="7419fe0f79001312aa562a10f8fc2bac288e410fc5ec752d4f832516cfe49c06"
Oct 30 04:00:08.414452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.405751    4877 scope.go:122] "RemoveContainer" containerID="2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb"
Oct 30 04:00:08.414452 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.406218    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk_odh-ai-gateway-infra(e24f3539-e5a7-4de6-9c42-ca52c8de6344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344"
Oct 30 04:00:08.477233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.476212    4877 generic.go:356] "Generic (PLEG): container finished" podID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerID="1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7" exitCode=60
Oct 30 04:00:08.485709 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.485668    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerDied","Data":"1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7"}
Oct 30 04:00:08.496586 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.487526    4877 scope.go:122] "RemoveContainer" containerID="1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7"
Oct 30 04:00:08.512444 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.497312    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls_odh-ai-gateway-infra(019067eb-aff1-42fe-9c71-7dcf9bcc29b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4"
Oct 30 04:00:08.512741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.505612    4877 scope.go:122] "RemoveContainer" containerID="83e5a88615296245a4876560a83b731f7638729ff072ffcd64cf950e62e58640"
Oct 30 04:00:08.533447 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.513364    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr_odh-ai-gateway-infra(e7409641-adac-4021-97f5-b3fcffb172a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" podUID="e7409641-adac-4021-97f5-b3fcffb172a8"
Oct 30 04:00:08.562763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.562723    4877 generic.go:356] "Generic (PLEG): container finished" podID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerID="4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e" exitCode=60
Oct 30 04:00:08.562966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.562906    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerDied","Data":"4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e"}
Oct 30 04:00:08.563240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.563215    4877 scope.go:122] "RemoveContainer" containerID="4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e"
Oct 30 04:00:08.564110 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.563708    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt_odh-ai-gateway-infra(989c1932-7a7a-4f64-93f5-6a46bd823d89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89"
Oct 30 04:00:08.568105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.568085    4877 scope.go:122] "RemoveContainer" containerID="55a85e7dc354128001229aa80125d3bb1a5c381b786937d24cddd794a18d7f4d"
Oct 30 04:00:08.569206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.569184    4877 generic.go:356] "Generic (PLEG): container finished" podID="836b510f-a531-4569-9bc1-3416e47cafee" containerID="35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6" exitCode=60
Oct 30 04:00:08.570914 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.569986    4877 scope.go:122] "RemoveContainer" containerID="35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6"
Oct 30 04:00:08.570914 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.570619    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n_odh-ai-gateway-infra(836b510f-a531-4569-9bc1-3416e47cafee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" podUID="836b510f-a531-4569-9bc1-3416e47cafee"
Oct 30 04:00:08.570914 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.570005    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerDied","Data":"35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6"}
Oct 30 04:00:08.573268 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.573244    4877 generic.go:356] "Generic (PLEG): container finished" podID="1e98105c-1514-413e-9bce-2cbc6761294a" containerID="081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f" exitCode=60
Oct 30 04:00:08.573638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.573321    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerDied","Data":"081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f"}
Oct 30 04:00:08.574115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.573626    4877 scope.go:122] "RemoveContainer" containerID="081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f"
Oct 30 04:00:08.575660 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.575608    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb_odh-ai-gateway-infra(1e98105c-1514-413e-9bce-2cbc6761294a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" podUID="1e98105c-1514-413e-9bce-2cbc6761294a"
Oct 30 04:00:08.576237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.576217    4877 generic.go:356] "Generic (PLEG): container finished" podID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerID="406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75" exitCode=60
Oct 30 04:00:08.576389 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.576301    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerDied","Data":"406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75"}
Oct 30 04:00:08.576816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.576586    4877 scope.go:122] "RemoveContainer" containerID="406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75"
Oct 30 04:00:08.578001 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.577974    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m_odh-ai-gateway-infra(d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9"
Oct 30 04:00:08.578743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.578724    4877 scope.go:122] "RemoveContainer" containerID="74ffa7c911b878404dbb238a3dbba57540ae4d023c6cd445827adf189e2be615"
Oct 30 04:00:08.580562 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.580536    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp_odh-ai-gateway-infra(008f7da2-1318-40d9-b589-bb94835510e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" podUID="008f7da2-1318-40d9-b589-bb94835510e6"
Oct 30 04:00:08.580657 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.580609    4877 generic.go:356] "Generic (PLEG): container finished" podID="e5316ebf-1c98-4b55-af72-243224166688" containerID="59c138d158ab9a6f852cc664965f4ea281e5fd3dd552ff119e97de6bd4446493" exitCode=60
Oct 30 04:00:08.580657 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.580639    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" event={"ID":"e5316ebf-1c98-4b55-af72-243224166688","Type":"ContainerDied","Data":"59c138d158ab9a6f852cc664965f4ea281e5fd3dd552ff119e97de6bd4446493"}
Oct 30 04:00:08.580935 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.580918    4877 scope.go:122] "RemoveContainer" containerID="59c138d158ab9a6f852cc664965f4ea281e5fd3dd552ff119e97de6bd4446493"
Oct 30 04:00:08.581165 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:08.581144    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl_odh-ai-gateway-infra(e5316ebf-1c98-4b55-af72-243224166688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" podUID="e5316ebf-1c98-4b55-af72-243224166688"
Oct 30 04:00:08.587239 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.587096    4877 scope.go:122] "RemoveContainer" containerID="d5dcb448c0a7bc09e60f08892561ce932ea522d18b37e5b62fb40ab9837d1a6f"
Oct 30 04:00:08.610885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.610847    4877 scope.go:122] "RemoveContainer" containerID="018b29ecc6b24aca38c06e5f032c4bab95918d718a2ed3af713afc7499348a3c"
Oct 30 04:00:08.646737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.646706    4877 scope.go:122] "RemoveContainer" containerID="b56b9613e98505f6920f3bfc3e9a322cadb068fdeb069eda6cce4fd3f04902b5"
Oct 30 04:00:08.665361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.665334    4877 scope.go:122] "RemoveContainer" containerID="954669dcdd37976e0aa4d1439d906530cacb93cfb03b08dc62377284a5334777"
Oct 30 04:00:08.681373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:08.681347    4877 scope.go:122] "RemoveContainer" containerID="75af4034906a966f36b6564df6d336de1bcea1561968c00da63c0a2e6e7b9f47"
Oct 30 04:00:09.593339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.593307    4877 scope.go:122] "RemoveContainer" containerID="1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7"
Oct 30 04:00:09.593832 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.593737    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls_odh-ai-gateway-infra(019067eb-aff1-42fe-9c71-7dcf9bcc29b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4"
Oct 30 04:00:09.595296 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.595274    4877 scope.go:122] "RemoveContainer" containerID="4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e"
Oct 30 04:00:09.595635 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.595608    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt_odh-ai-gateway-infra(989c1932-7a7a-4f64-93f5-6a46bd823d89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89"
Oct 30 04:00:09.597080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.597059    4877 scope.go:122] "RemoveContainer" containerID="35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6"
Oct 30 04:00:09.597360 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.597339    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n_odh-ai-gateway-infra(836b510f-a531-4569-9bc1-3416e47cafee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" podUID="836b510f-a531-4569-9bc1-3416e47cafee"
Oct 30 04:00:09.598691 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.598672    4877 scope.go:122] "RemoveContainer" containerID="081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f"
Oct 30 04:00:09.598931 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.598911    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb_odh-ai-gateway-infra(1e98105c-1514-413e-9bce-2cbc6761294a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" podUID="1e98105c-1514-413e-9bce-2cbc6761294a"
Oct 30 04:00:09.600285 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.600268    4877 scope.go:122] "RemoveContainer" containerID="406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75"
Oct 30 04:00:09.600540 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.600520    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m_odh-ai-gateway-infra(d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9"
Oct 30 04:00:09.601946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.601924    4877 scope.go:122] "RemoveContainer" containerID="59c138d158ab9a6f852cc664965f4ea281e5fd3dd552ff119e97de6bd4446493"
Oct 30 04:00:09.602204 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.602181    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl_odh-ai-gateway-infra(e5316ebf-1c98-4b55-af72-243224166688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" podUID="e5316ebf-1c98-4b55-af72-243224166688"
Oct 30 04:00:09.603406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:09.603390    4877 scope.go:122] "RemoveContainer" containerID="2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb"
Oct 30 04:00:09.603643 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:09.603624    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk_odh-ai-gateway-infra(e24f3539-e5a7-4de6-9c42-ca52c8de6344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344"
Oct 30 04:00:11.028736 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:11.028701    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"]
Oct 30 04:00:11.038010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:11.037982    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7c8396-l9vrz"]
Oct 30 04:00:11.706031 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:11.705994    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="781be59f-c261-49e7-a461-ff1ba0fa0c72" path="/var/lib/kubelet/pods/781be59f-c261-49e7-a461-ff1ba0fa0c72/volumes"
Oct 30 04:00:13.039308 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:13.039272    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"]
Oct 30 04:00:13.075492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:13.075448    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"]
Oct 30 04:00:16.231800 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:16.231757    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:00:16.691215 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:16.691187    4877 scope.go:122] "RemoveContainer" containerID="86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329"
Oct 30 04:00:16.691737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:16.691703    4877 scope.go:122] "RemoveContainer" containerID="edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0"
Oct 30 04:00:17.069279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.069240    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"]
Oct 30 04:00:17.401831 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.398747    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:17.559191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.546517    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr\" (UniqueName: \"kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr\") pod \"e5316ebf-1c98-4b55-af72-243224166688\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") "
Oct 30 04:00:17.559191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.546598    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca\") pod \"e5316ebf-1c98-4b55-af72-243224166688\" (UID: \"e5316ebf-1c98-4b55-af72-243224166688\") "
Oct 30 04:00:17.584535 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.572694    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca" pod "e5316ebf-1c98-4b55-af72-243224166688" (UID: "e5316ebf-1c98-4b55-af72-243224166688"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:17.629459 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.625729    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr" pod "e5316ebf-1c98-4b55-af72-243224166688" (UID: "e5316ebf-1c98-4b55-af72-243224166688"). InnerVolumeSpecName "kube-api-access-xlvhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:17.661251 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.661213    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlvhr\" (UniqueName: \"kubernetes.io/projected/e5316ebf-1c98-4b55-af72-243224166688-kube-api-access-xlvhr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:17.661474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.661458    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5316ebf-1c98-4b55-af72-243224166688-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:17.717285 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.716745    4877 scope.go:122] "RemoveContainer" containerID="4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e"
Oct 30 04:00:17.754255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.754222    4877 scope.go:122] "RemoveContainer" containerID="3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4"
Oct 30 04:00:17.770275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.770247    4877 scope.go:122] "RemoveContainer" containerID="35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6"
Oct 30 04:00:17.772564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.772541    4877 scope.go:122] "RemoveContainer" containerID="1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7"
Oct 30 04:00:17.775221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.775167    4877 scope.go:122] "RemoveContainer" containerID="e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a"
Oct 30 04:00:17.785743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.785528    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl" event={"ID":"e5316ebf-1c98-4b55-af72-243224166688","Type":"ContainerDied","Data":"0c88fc08c65d545baea77ecf506d4ced74abebe040429c821cc4fe9407d8bc00"}
Oct 30 04:00:17.785945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.785923    4877 scope.go:122] "RemoveContainer" containerID="59c138d158ab9a6f852cc664965f4ea281e5fd3dd552ff119e97de6bd4446493"
Oct 30 04:00:17.786184 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.786169    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"
Oct 30 04:00:17.838836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.838800    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerStarted","Data":"a613f62e7bcddf7b09a6d892f9f13d04ca0887cc8897e7a2b8abb1f672ce81b8"}
Oct 30 04:00:17.844342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.844313    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerStarted","Data":"450df28c18a691488388b26c412eaee773d585bbb2e0b46a33870cd2d076cdf0"}
Oct 30 04:00:17.845918 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.845893    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"]
Oct 30 04:00:17.855354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:17.855317    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9e0d09-29888880-fmmkl"]
Oct 30 04:00:18.041525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.041474    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" podUID="a24f7bc8-24b0-463b-9667-450610471d42" containerName="payload-processing" containerID="cri-o://ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065" gracePeriod=25
Oct 30 04:00:18.076635 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.076490    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerName="payload-pre-processing" containerID="cri-o://ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279" gracePeriod=25
Oct 30 04:00:18.691247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.691163    4877 scope.go:122] "RemoveContainer" containerID="a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323"
Oct 30 04:00:18.691712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.691364    4877 scope.go:122] "RemoveContainer" containerID="74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd"
Oct 30 04:00:18.691804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.691724    4877 scope.go:122] "RemoveContainer" containerID="83e5a88615296245a4876560a83b731f7638729ff072ffcd64cf950e62e58640"
Oct 30 04:00:18.691936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.691916    4877 scope.go:122] "RemoveContainer" containerID="4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f"
Oct 30 04:00:18.692080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.692056    4877 scope.go:122] "RemoveContainer" containerID="e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2"
Oct 30 04:00:18.692467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.692448    4877 scope.go:122] "RemoveContainer" containerID="74ffa7c911b878404dbb238a3dbba57540ae4d023c6cd445827adf189e2be615"
Oct 30 04:00:18.855982 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.855939    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerStarted","Data":"bbe8bcf6d8f3a02bbd859793f918c31b597f863933063094162adfd14a7887dc"}
Oct 30 04:00:18.862791 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.862762    4877 generic.go:356] "Generic (PLEG): container finished" podID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerID="a613f62e7bcddf7b09a6d892f9f13d04ca0887cc8897e7a2b8abb1f672ce81b8" exitCode=60
Oct 30 04:00:18.862915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.862857    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerDied","Data":"a613f62e7bcddf7b09a6d892f9f13d04ca0887cc8897e7a2b8abb1f672ce81b8"}
Oct 30 04:00:18.862915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.862906    4877 scope.go:122] "RemoveContainer" containerID="86adbb385fb0d79c85b4155c1ff62dab3b65ad76b0de51c87a95bc79b6f0a329"
Oct 30 04:00:18.863362 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.863346    4877 scope.go:122] "RemoveContainer" containerID="a613f62e7bcddf7b09a6d892f9f13d04ca0887cc8897e7a2b8abb1f672ce81b8"
Oct 30 04:00:18.863802 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:18.863779    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86_odh-ai-gateway-infra(aa661a42-aaa6-4ff8-947f-3962a969dac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8"
Oct 30 04:00:18.867158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.867131    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerStarted","Data":"7a2ff496408c09265a13541f34e5d6add5d5e6ca6bfebbe686bed28da0fa98d1"}
Oct 30 04:00:18.869199 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.869178    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerStarted","Data":"48850f624f15b8cb4aa2a17edceae10f390032a91cec9a497a18dc4964fcb45e"}
Oct 30 04:00:18.871206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.871185    4877 generic.go:356] "Generic (PLEG): container finished" podID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerID="450df28c18a691488388b26c412eaee773d585bbb2e0b46a33870cd2d076cdf0" exitCode=60
Oct 30 04:00:18.871297 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.871274    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerDied","Data":"450df28c18a691488388b26c412eaee773d585bbb2e0b46a33870cd2d076cdf0"}
Oct 30 04:00:18.871698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.871674    4877 scope.go:122] "RemoveContainer" containerID="450df28c18a691488388b26c412eaee773d585bbb2e0b46a33870cd2d076cdf0"
Oct 30 04:00:18.872006 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:18.871980    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt_odh-ai-gateway-infra(f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd"
Oct 30 04:00:18.873642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.873622    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerStarted","Data":"605b31839cedf7a4da07e50dd8f7b72bf25d611c299a32036535a09cda0ed5ed"}
Oct 30 04:00:18.875696 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.875673    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerStarted","Data":"97a6498e068dad060d7f68d1297201123deb3d43e431271cfacbd3983e412b63"}
Oct 30 04:00:18.891518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.891275    4877 scope.go:122] "RemoveContainer" containerID="edc5fc1974cd170cececb93e316f41d26f94e3a1755afc4912bed0d45c4961b0"
Oct 30 04:00:18.903321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.903299    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"]
Oct 30 04:00:18.927660 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:18.927631    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"]
Oct 30 04:00:19.691566 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.691533    4877 scope.go:122] "RemoveContainer" containerID="406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75"
Oct 30 04:00:19.692698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.692637    4877 scope.go:122] "RemoveContainer" containerID="71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6"
Oct 30 04:00:19.696512 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.693058    4877 scope.go:122] "RemoveContainer" containerID="20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb"
Oct 30 04:00:19.696512 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.693979    4877 scope.go:122] "RemoveContainer" containerID="2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb"
Oct 30 04:00:19.696512 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.694815    4877 scope.go:122] "RemoveContainer" containerID="a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df"
Oct 30 04:00:19.718289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.718260    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5316ebf-1c98-4b55-af72-243224166688" path="/var/lib/kubelet/pods/e5316ebf-1c98-4b55-af72-243224166688/volumes"
Oct 30 04:00:19.895951 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.895922    4877 generic.go:356] "Generic (PLEG): container finished" podID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerID="97a6498e068dad060d7f68d1297201123deb3d43e431271cfacbd3983e412b63" exitCode=60
Oct 30 04:00:19.896131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.895995    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerDied","Data":"97a6498e068dad060d7f68d1297201123deb3d43e431271cfacbd3983e412b63"}
Oct 30 04:00:19.896131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.896067    4877 scope.go:122] "RemoveContainer" containerID="4cea40ba95108b1bd707cc68962f782fa0ca8850059afb1e98f29ece01492c6e"
Oct 30 04:00:19.896526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.896509    4877 scope.go:122] "RemoveContainer" containerID="97a6498e068dad060d7f68d1297201123deb3d43e431271cfacbd3983e412b63"
Oct 30 04:00:19.896900 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:19.896882    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt_odh-ai-gateway-infra(989c1932-7a7a-4f64-93f5-6a46bd823d89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89"
Oct 30 04:00:19.899644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.899622    4877 generic.go:356] "Generic (PLEG): container finished" podID="836b510f-a531-4569-9bc1-3416e47cafee" containerID="bbe8bcf6d8f3a02bbd859793f918c31b597f863933063094162adfd14a7887dc" exitCode=60
Oct 30 04:00:19.899770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.899709    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerDied","Data":"bbe8bcf6d8f3a02bbd859793f918c31b597f863933063094162adfd14a7887dc"}
Oct 30 04:00:19.900142 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.900122    4877 scope.go:122] "RemoveContainer" containerID="bbe8bcf6d8f3a02bbd859793f918c31b597f863933063094162adfd14a7887dc"
Oct 30 04:00:19.900618 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:19.900589    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n_odh-ai-gateway-infra(836b510f-a531-4569-9bc1-3416e47cafee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" podUID="836b510f-a531-4569-9bc1-3416e47cafee"
Oct 30 04:00:19.902474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.902410    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerStarted","Data":"2195dc81f77e80fd7da0ae09e3f61f57fc7b71edbbedd57600bd99e324fd8361"}
Oct 30 04:00:19.905347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.905324    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerStarted","Data":"7fe32821d179f607e76146bee8e55110280ab8d010f35d28c601324fbbea4b96"}
Oct 30 04:00:19.907313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.907294    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"]
Oct 30 04:00:19.908704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.908680    4877 generic.go:356] "Generic (PLEG): container finished" podID="008f7da2-1318-40d9-b589-bb94835510e6" containerID="93f2371d54b412e94dbceb779eeba987189da7d402bd7a5002f718d75aafeb05" exitCode=60
Oct 30 04:00:19.908820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.908804    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" event={"ID":"008f7da2-1318-40d9-b589-bb94835510e6","Type":"ContainerDied","Data":"93f2371d54b412e94dbceb779eeba987189da7d402bd7a5002f718d75aafeb05"}
Oct 30 04:00:19.918558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.909212    4877 scope.go:122] "RemoveContainer" containerID="93f2371d54b412e94dbceb779eeba987189da7d402bd7a5002f718d75aafeb05"
Oct 30 04:00:19.918558 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:19.912450    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp_odh-ai-gateway-infra(008f7da2-1318-40d9-b589-bb94835510e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" podUID="008f7da2-1318-40d9-b589-bb94835510e6"
Oct 30 04:00:19.918558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.916250    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerStarted","Data":"6687e9389d224fff2fbccae55ffe23c5267f45f2ef4f7b87a920ded815e130bc"}
Oct 30 04:00:19.920550 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.920521    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerStarted","Data":"d18ef083b529c964386a8ccd12631544be988ce2ab7e2fa5235549f2e7e6bc74"}
Oct 30 04:00:19.921766 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.921698    4877 scope.go:122] "RemoveContainer" containerID="35ef8d27f9edd4e71954e4a33a6023562b11e9b40d0bfffccc3d5ba75d123fc6"
Oct 30 04:00:19.946132 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.946105    4877 generic.go:356] "Generic (PLEG): container finished" podID="fe99e906-f412-4f34-91d1-c00607c1f067" containerID="7a2ff496408c09265a13541f34e5d6add5d5e6ca6bfebbe686bed28da0fa98d1" exitCode=60
Oct 30 04:00:19.946274 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.946205    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerDied","Data":"7a2ff496408c09265a13541f34e5d6add5d5e6ca6bfebbe686bed28da0fa98d1"}
Oct 30 04:00:19.946611 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.946593    4877 scope.go:122] "RemoveContainer" containerID="7a2ff496408c09265a13541f34e5d6add5d5e6ca6bfebbe686bed28da0fa98d1"
Oct 30 04:00:19.946903 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:19.946880    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c_odh-ai-gateway-infra(fe99e906-f412-4f34-91d1-c00607c1f067)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" podUID="fe99e906-f412-4f34-91d1-c00607c1f067"
Oct 30 04:00:19.949161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.949140    4877 generic.go:356] "Generic (PLEG): container finished" podID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerID="48850f624f15b8cb4aa2a17edceae10f390032a91cec9a497a18dc4964fcb45e" exitCode=60
Oct 30 04:00:19.949298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.949280    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerDied","Data":"48850f624f15b8cb4aa2a17edceae10f390032a91cec9a497a18dc4964fcb45e"}
Oct 30 04:00:19.949772 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.949750    4877 scope.go:122] "RemoveContainer" containerID="48850f624f15b8cb4aa2a17edceae10f390032a91cec9a497a18dc4964fcb45e"
Oct 30 04:00:19.950186 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:19.950149    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8_odh-ai-gateway-infra(ad2f1fa7-7466-4640-935e-f78f3aa91ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5"
Oct 30 04:00:19.954218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.954201    4877 scope.go:122] "RemoveContainer" containerID="74ffa7c911b878404dbb238a3dbba57540ae4d023c6cd445827adf189e2be615"
Oct 30 04:00:19.956155 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.956132    4877 generic.go:356] "Generic (PLEG): container finished" podID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerID="605b31839cedf7a4da07e50dd8f7b72bf25d611c299a32036535a09cda0ed5ed" exitCode=60
Oct 30 04:00:19.956243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.956229    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerDied","Data":"605b31839cedf7a4da07e50dd8f7b72bf25d611c299a32036535a09cda0ed5ed"}
Oct 30 04:00:19.956533 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.956514    4877 scope.go:122] "RemoveContainer" containerID="605b31839cedf7a4da07e50dd8f7b72bf25d611c299a32036535a09cda0ed5ed"
Oct 30 04:00:19.956779 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:19.956762    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls_odh-ai-gateway-infra(019067eb-aff1-42fe-9c71-7dcf9bcc29b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4"
Oct 30 04:00:19.958909 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.958888    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" event={"ID":"e7409641-adac-4021-97f5-b3fcffb172a8","Type":"ContainerStarted","Data":"384870fd756d2568d162d02c1f737d7b0060ef250a6320bd0f3578b56e86279c"}
Oct 30 04:00:19.970783 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:19.970762    4877 scope.go:122] "RemoveContainer" containerID="e8d2122f753f4e23584c1384315de2bcbc50246b8d0aa1ff1ad80c8cb4127f2a"
Oct 30 04:00:20.047891 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.047859    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"]
Oct 30 04:00:20.070290 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.067277    4877 scope.go:122] "RemoveContainer" containerID="3171ed15341d06da4696901ffb31d27de98c1f2e390e121729e4f5b787c616a4"
Oct 30 04:00:20.078570 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.078477    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"]
Oct 30 04:00:20.138021 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.137865    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"]
Oct 30 04:00:20.147022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.146990    4877 scope.go:122] "RemoveContainer" containerID="1c8c1ef34a408f4f85c4c1f6ac8451d6be661c688b2f2ad5089ba9f65f825ae7"
Oct 30 04:00:20.226103 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.226065    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"]
Oct 30 04:00:20.280989 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.280842    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:20.326418 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.326356    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" podStartSLOduration=20.326338192 podStartE2EDuration="20.326338192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:20.318912461 +0000 UTC m=+3448.747801879" watchObservedRunningTime="2026-10-30 04:00:20.326338192 +0000 UTC m=+3448.755227610"
Oct 30 04:00:20.344617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.344593    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca\") pod \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") "
Oct 30 04:00:20.344739 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.344645    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr\" (UniqueName: \"kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr\") pod \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\" (UID: \"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd\") "
Oct 30 04:00:20.345013 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.344990    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca" pod "f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" (UID: "f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:20.345334 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.345316    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:20.346883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.346860    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr" pod "f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" (UID: "f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd"). InnerVolumeSpecName "kube-api-access-x67sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:20.416648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.416572    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:20.446486 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.446416    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca\") pod \"aa661a42-aaa6-4ff8-947f-3962a969dac8\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") "
Oct 30 04:00:20.446669 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.446642    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t\" (UniqueName: \"kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t\") pod \"aa661a42-aaa6-4ff8-947f-3962a969dac8\" (UID: \"aa661a42-aaa6-4ff8-947f-3962a969dac8\") "
Oct 30 04:00:20.446826 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.446803    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca" pod "aa661a42-aaa6-4ff8-947f-3962a969dac8" (UID: "aa661a42-aaa6-4ff8-947f-3962a969dac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:20.447344 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.447317    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa661a42-aaa6-4ff8-947f-3962a969dac8-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:20.447344 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.447347    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67sr\" (UniqueName: \"kubernetes.io/projected/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd-kube-api-access-x67sr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:20.449056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.449029    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t" pod "aa661a42-aaa6-4ff8-947f-3962a969dac8" (UID: "aa661a42-aaa6-4ff8-947f-3962a969dac8"). InnerVolumeSpecName "kube-api-access-q7r9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:20.549513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.549466    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7r9t\" (UniqueName: \"kubernetes.io/projected/aa661a42-aaa6-4ff8-947f-3962a969dac8-kube-api-access-q7r9t\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:20.691039 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.690963    4877 scope.go:122] "RemoveContainer" containerID="081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f"
Oct 30 04:00:20.977532 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.977489    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerID="2195dc81f77e80fd7da0ae09e3f61f57fc7b71edbbedd57600bd99e324fd8361" exitCode=60
Oct 30 04:00:20.978121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.977604    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerDied","Data":"2195dc81f77e80fd7da0ae09e3f61f57fc7b71edbbedd57600bd99e324fd8361"}
Oct 30 04:00:20.978121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.977662    4877 scope.go:122] "RemoveContainer" containerID="4bb616a8a24bf46c8dc011e5ae22b78e65dfad959344c24950fd1a943e584a6f"
Oct 30 04:00:20.979020 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.978903    4877 scope.go:122] "RemoveContainer" containerID="2195dc81f77e80fd7da0ae09e3f61f57fc7b71edbbedd57600bd99e324fd8361"
Oct 30 04:00:20.979660 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:20.979342    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6_odh-ai-gateway-infra(e7a919ba-751f-4778-8d8c-7cd445d1c3d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2"
Oct 30 04:00:20.983330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.983305    4877 generic.go:356] "Generic (PLEG): container finished" podID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerID="7fe32821d179f607e76146bee8e55110280ab8d010f35d28c601324fbbea4b96" exitCode=60
Oct 30 04:00:20.983455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.983436    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerDied","Data":"7fe32821d179f607e76146bee8e55110280ab8d010f35d28c601324fbbea4b96"}
Oct 30 04:00:20.983955 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.983931    4877 scope.go:122] "RemoveContainer" containerID="7fe32821d179f607e76146bee8e55110280ab8d010f35d28c601324fbbea4b96"
Oct 30 04:00:20.984518 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:20.984491    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc_odh-ai-gateway-infra(e81dceac-bb98-4f7c-9995-d91a2fef84d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0"
Oct 30 04:00:20.991073 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.991044    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"]
Oct 30 04:00:20.992289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.991200    4877 generic.go:356] "Generic (PLEG): container finished" podID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerID="d1c6c16128c1ccfdeca97336fac92cee7b22c2a427197eafd0379542cb287ea1" exitCode=60
Oct 30 04:00:20.992289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.991307    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerDied","Data":"d1c6c16128c1ccfdeca97336fac92cee7b22c2a427197eafd0379542cb287ea1"}
Oct 30 04:00:20.992289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:20.991814    4877 scope.go:122] "RemoveContainer" containerID="d1c6c16128c1ccfdeca97336fac92cee7b22c2a427197eafd0379542cb287ea1"
Oct 30 04:00:20.992289 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:20.992240    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m_odh-ai-gateway-infra(d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9"
Oct 30 04:00:21.001380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.000413    4877 generic.go:356] "Generic (PLEG): container finished" podID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerID="94799d88d9646575dbe446532f0a451655498fe4cb87493d7c1e9309966555ca" exitCode=60
Oct 30 04:00:21.001380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.000549    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" event={"ID":"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c","Type":"ContainerDied","Data":"94799d88d9646575dbe446532f0a451655498fe4cb87493d7c1e9309966555ca"}
Oct 30 04:00:21.001380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.000939    4877 scope.go:122] "RemoveContainer" containerID="94799d88d9646575dbe446532f0a451655498fe4cb87493d7c1e9309966555ca"
Oct 30 04:00:21.001380 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.001324    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8_odh-ai-gateway-infra(5331a17c-1fa3-46ae-bba8-ec3d5f94f68c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c"
Oct 30 04:00:21.005554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.004141    4877 generic.go:356] "Generic (PLEG): container finished" podID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerID="6687e9389d224fff2fbccae55ffe23c5267f45f2ef4f7b87a920ded815e130bc" exitCode=60
Oct 30 04:00:21.005554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.004257    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerDied","Data":"6687e9389d224fff2fbccae55ffe23c5267f45f2ef4f7b87a920ded815e130bc"}
Oct 30 04:00:21.005554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.004713    4877 scope.go:122] "RemoveContainer" containerID="6687e9389d224fff2fbccae55ffe23c5267f45f2ef4f7b87a920ded815e130bc"
Oct 30 04:00:21.005554 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.005125    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms_odh-ai-gateway-infra(7ced66d5-2223-4af9-9a6d-9de05d1f6750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750"
Oct 30 04:00:21.026672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.019271    4877 generic.go:356] "Generic (PLEG): container finished" podID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerID="d18ef083b529c964386a8ccd12631544be988ce2ab7e2fa5235549f2e7e6bc74" exitCode=60
Oct 30 04:00:21.026672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.019360    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerDied","Data":"d18ef083b529c964386a8ccd12631544be988ce2ab7e2fa5235549f2e7e6bc74"}
Oct 30 04:00:21.026672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.020180    4877 scope.go:122] "RemoveContainer" containerID="d18ef083b529c964386a8ccd12631544be988ce2ab7e2fa5235549f2e7e6bc74"
Oct 30 04:00:21.026672 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.020818    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n_odh-ai-gateway-infra(d9413205-622e-4cee-82f9-e6c9fd2041fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe"
Oct 30 04:00:21.030069 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.028924    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerStarted","Data":"fd44839b463d2c485fde34799f38a77179da879acc7ce44df584b0f91eb6f848"}
Oct 30 04:00:21.030069 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.029518    4877 scope.go:122] "RemoveContainer" containerID="fd44839b463d2c485fde34799f38a77179da879acc7ce44df584b0f91eb6f848"
Oct 30 04:00:21.030704 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.030659    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk_odh-ai-gateway-infra(e24f3539-e5a7-4de6-9c42-ca52c8de6344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344"
Oct 30 04:00:21.031473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.031103    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86" event={"ID":"aa661a42-aaa6-4ff8-947f-3962a969dac8","Type":"ContainerDied","Data":"dc6ca39ac64292c6bed794ea0a09c4fc5d2d53bf0fec83b0107591888bd697e1"}
Oct 30 04:00:21.031473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.031181    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"
Oct 30 04:00:21.034526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.034361    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerStarted","Data":"a30c9392c185e3e345de90002d15687df5f250b1508575aae3c9c7cb25964c88"}
Oct 30 04:00:21.035003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.034874    4877 scope.go:122] "RemoveContainer" containerID="a30c9392c185e3e345de90002d15687df5f250b1508575aae3c9c7cb25964c88"
Oct 30 04:00:21.035240 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.035216    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd_odh-ai-gateway-infra(eaf0d3e6-b75e-4652-a4f1-a348c357dc9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e"
Oct 30 04:00:21.035751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.035711    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"]
Oct 30 04:00:21.049615 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.049588    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerStarted","Data":"28637b62a217cc834dc41d1b2f3ae1b95bf6e9ce7ba834212036fa782d5befa7"}
Oct 30 04:00:21.050022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.050003    4877 scope.go:122] "RemoveContainer" containerID="28637b62a217cc834dc41d1b2f3ae1b95bf6e9ce7ba834212036fa782d5befa7"
Oct 30 04:00:21.050415 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.050392    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh_odh-ai-gateway-infra(722060d3-d60d-4c96-972d-71e3b1408128)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" podUID="722060d3-d60d-4c96-972d-71e3b1408128"
Oct 30 04:00:21.070003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.054971    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt" event={"ID":"f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd","Type":"ContainerDied","Data":"9b93d13f0ff30c09c4dee0332e20d256f11237d79bfad56bd803d03e8d4480d0"}
Oct 30 04:00:21.070003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.055058    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"
Oct 30 04:00:21.099543 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.090811    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7409641-adac-4021-97f5-b3fcffb172a8" containerID="384870fd756d2568d162d02c1f737d7b0060ef250a6320bd0f3578b56e86279c" exitCode=60
Oct 30 04:00:21.099543 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.090864    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" event={"ID":"e7409641-adac-4021-97f5-b3fcffb172a8","Type":"ContainerDied","Data":"384870fd756d2568d162d02c1f737d7b0060ef250a6320bd0f3578b56e86279c"}
Oct 30 04:00:21.099543 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.091305    4877 scope.go:122] "RemoveContainer" containerID="384870fd756d2568d162d02c1f737d7b0060ef250a6320bd0f3578b56e86279c"
Oct 30 04:00:21.099543 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:21.091764    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr_odh-ai-gateway-infra(e7409641-adac-4021-97f5-b3fcffb172a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" podUID="e7409641-adac-4021-97f5-b3fcffb172a8"
Oct 30 04:00:21.130925 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.130830    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"]
Oct 30 04:00:21.161577 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.161480    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"]
Oct 30 04:00:21.183493 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.180539    4877 scope.go:122] "RemoveContainer" containerID="74229e44cd2d8b4a7f8e33e41493c18ca32734017bff2bf2026a09cf4ce555fd"
Oct 30 04:00:21.193740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.192245    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"]
Oct 30 04:00:21.212574 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.212204    4877 scope.go:122] "RemoveContainer" containerID="406b21452505b93478124f54f6bb1e048610ddf3928e9ed1cc532105ab3a1a75"
Oct 30 04:00:21.459446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.453677    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"]
Oct 30 04:00:21.502449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.486668    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8d80b1-29888880-spnvt"]
Oct 30 04:00:21.502449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.488315    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"]
Oct 30 04:00:21.559722 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.558721    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:21.603117 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.603089    4877 scope.go:122] "RemoveContainer" containerID="20f9f3907b5b63759b3ecf50a245f87014a7cf2980547add0ce5ec13536011cb"
Oct 30 04:00:21.648558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.648528    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:21.670401 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.670362    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca\") pod \"836b510f-a531-4569-9bc1-3416e47cafee\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") "
Oct 30 04:00:21.670913 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.670889    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn\" (UniqueName: \"kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn\") pod \"836b510f-a531-4569-9bc1-3416e47cafee\" (UID: \"836b510f-a531-4569-9bc1-3416e47cafee\") "
Oct 30 04:00:21.679199 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.679174    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca" pod "836b510f-a531-4569-9bc1-3416e47cafee" (UID: "836b510f-a531-4569-9bc1-3416e47cafee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:21.689464 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.689407    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn" pod "836b510f-a531-4569-9bc1-3416e47cafee" (UID: "836b510f-a531-4569-9bc1-3416e47cafee"). InnerVolumeSpecName "kube-api-access-kjgxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:21.725744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.725404    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" path="/var/lib/kubelet/pods/f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd/volumes"
Oct 30 04:00:21.743230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.742281    4877 scope.go:122] "RemoveContainer" containerID="e067eb092a4a870b19441bf9a0674849a92b46ccd753a2c5ef21a8ee430575d2"
Oct 30 04:00:21.776342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.775536    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9\" (UniqueName: \"kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9\") pod \"989c1932-7a7a-4f64-93f5-6a46bd823d89\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") "
Oct 30 04:00:21.776342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.775892    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca\") pod \"989c1932-7a7a-4f64-93f5-6a46bd823d89\" (UID: \"989c1932-7a7a-4f64-93f5-6a46bd823d89\") "
Oct 30 04:00:21.777249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.777226    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjgxn\" (UniqueName: \"kubernetes.io/projected/836b510f-a531-4569-9bc1-3416e47cafee-kube-api-access-kjgxn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.777402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.777388    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836b510f-a531-4569-9bc1-3416e47cafee-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.778129 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.778063    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca" pod "989c1932-7a7a-4f64-93f5-6a46bd823d89" (UID: "989c1932-7a7a-4f64-93f5-6a46bd823d89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:21.792500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.791998    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9" pod "989c1932-7a7a-4f64-93f5-6a46bd823d89" (UID: "989c1932-7a7a-4f64-93f5-6a46bd823d89"). InnerVolumeSpecName "kube-api-access-rlcd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:21.801494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.801406    4877 scope.go:122] "RemoveContainer" containerID="a4fe6abbba96141f5db45cc8a6f004dac4157ebc8100edc4a92713019f908323"
Oct 30 04:00:21.841622 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.841589    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"]
Oct 30 04:00:21.849846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.849813    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-35f43f-29888880-qds86"]
Oct 30 04:00:21.868053 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.868024    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:21.879708 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.879677    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/989c1932-7a7a-4f64-93f5-6a46bd823d89-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.879996 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.879976    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlcd9\" (UniqueName: \"kubernetes.io/projected/989c1932-7a7a-4f64-93f5-6a46bd823d89-kube-api-access-rlcd9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.881675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.881651    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:21.883249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.883222    4877 scope.go:122] "RemoveContainer" containerID="a613f62e7bcddf7b09a6d892f9f13d04ca0887cc8897e7a2b8abb1f672ce81b8"
Oct 30 04:00:21.885292 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.885271    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:21.923797 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.923771    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:21.924072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.924014    4877 scope.go:122] "RemoveContainer" containerID="450df28c18a691488388b26c412eaee773d585bbb2e0b46a33870cd2d076cdf0"
Oct 30 04:00:21.935344 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.935322    4877 scope.go:122] "RemoveContainer" containerID="83e5a88615296245a4876560a83b731f7638729ff072ffcd64cf950e62e58640"
Oct 30 04:00:21.981112 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981072    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca\") pod \"fe99e906-f412-4f34-91d1-c00607c1f067\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") "
Oct 30 04:00:21.981541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981169    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq\" (UniqueName: \"kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq\") pod \"fe99e906-f412-4f34-91d1-c00607c1f067\" (UID: \"fe99e906-f412-4f34-91d1-c00607c1f067\") "
Oct 30 04:00:21.981541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981372    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w\" (UniqueName: \"kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w\") pod \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") "
Oct 30 04:00:21.981541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981497    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca" pod "fe99e906-f412-4f34-91d1-c00607c1f067" (UID: "fe99e906-f412-4f34-91d1-c00607c1f067"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:21.981541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981523    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca\") pod \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\" (UID: \"019067eb-aff1-42fe-9c71-7dcf9bcc29b4\") "
Oct 30 04:00:21.981743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981637    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca\") pod \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") "
Oct 30 04:00:21.981812 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981758    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5\" (UniqueName: \"kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5\") pod \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\" (UID: \"ad2f1fa7-7466-4640-935e-f78f3aa91ad5\") "
Oct 30 04:00:21.981949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981928    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca" pod "019067eb-aff1-42fe-9c71-7dcf9bcc29b4" (UID: "019067eb-aff1-42fe-9c71-7dcf9bcc29b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:21.982038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.981949    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca" pod "ad2f1fa7-7466-4640-935e-f78f3aa91ad5" (UID: "ad2f1fa7-7466-4640-935e-f78f3aa91ad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:21.982608 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.982584    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe99e906-f412-4f34-91d1-c00607c1f067-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.982770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.982613    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.982770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.982629    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:21.983878 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.983852    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq" pod "fe99e906-f412-4f34-91d1-c00607c1f067" (UID: "fe99e906-f412-4f34-91d1-c00607c1f067"). InnerVolumeSpecName "kube-api-access-kh4nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:21.984138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.984114    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w" pod "019067eb-aff1-42fe-9c71-7dcf9bcc29b4" (UID: "019067eb-aff1-42fe-9c71-7dcf9bcc29b4"). InnerVolumeSpecName "kube-api-access-p8r4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:21.984301 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:21.984282    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5" pod "ad2f1fa7-7466-4640-935e-f78f3aa91ad5" (UID: "ad2f1fa7-7466-4640-935e-f78f3aa91ad5"). InnerVolumeSpecName "kube-api-access-zp4t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.086279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.086159    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca\") pod \"008f7da2-1318-40d9-b589-bb94835510e6\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") "
Oct 30 04:00:22.086698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.086544    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8\" (UniqueName: \"kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8\") pod \"008f7da2-1318-40d9-b589-bb94835510e6\" (UID: \"008f7da2-1318-40d9-b589-bb94835510e6\") "
Oct 30 04:00:22.086804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.086723    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca" pod "008f7da2-1318-40d9-b589-bb94835510e6" (UID: "008f7da2-1318-40d9-b589-bb94835510e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.087503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.087337    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008f7da2-1318-40d9-b589-bb94835510e6-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.087503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.087367    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8r4w\" (UniqueName: \"kubernetes.io/projected/019067eb-aff1-42fe-9c71-7dcf9bcc29b4-kube-api-access-p8r4w\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.087503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.087383    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4t5\" (UniqueName: \"kubernetes.io/projected/ad2f1fa7-7466-4640-935e-f78f3aa91ad5-kube-api-access-zp4t5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.087503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.087398    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4nq\" (UniqueName: \"kubernetes.io/projected/fe99e906-f412-4f34-91d1-c00607c1f067-kube-api-access-kh4nq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.090328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.090273    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8" pod "008f7da2-1318-40d9-b589-bb94835510e6" (UID: "008f7da2-1318-40d9-b589-bb94835510e6"). InnerVolumeSpecName "kube-api-access-clhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.119997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.119929    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"]
Oct 30 04:00:22.134584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.134095    4877 generic.go:356] "Generic (PLEG): container finished" podID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerID="fd44839b463d2c485fde34799f38a77179da879acc7ce44df584b0f91eb6f848" exitCode=60
Oct 30 04:00:22.134747 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.134717    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerDied","Data":"fd44839b463d2c485fde34799f38a77179da879acc7ce44df584b0f91eb6f848"}
Oct 30 04:00:22.134969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.134949    4877 scope.go:122] "RemoveContainer" containerID="2b2f94b9c55e0647f7960ce1c9905c2fd91581eac5b7571d2bf376008c64dafb"
Oct 30 04:00:22.140224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.136195    4877 scope.go:122] "RemoveContainer" containerID="fd44839b463d2c485fde34799f38a77179da879acc7ce44df584b0f91eb6f848"
Oct 30 04:00:22.140224 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:22.136658    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk_odh-ai-gateway-infra(e24f3539-e5a7-4de6-9c42-ca52c8de6344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344"
Oct 30 04:00:22.143403 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.143375    4877 generic.go:356] "Generic (PLEG): container finished" podID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerID="a30c9392c185e3e345de90002d15687df5f250b1508575aae3c9c7cb25964c88" exitCode=60
Oct 30 04:00:22.143571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.143515    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerDied","Data":"a30c9392c185e3e345de90002d15687df5f250b1508575aae3c9c7cb25964c88"}
Oct 30 04:00:22.143986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.143967    4877 scope.go:122] "RemoveContainer" containerID="a30c9392c185e3e345de90002d15687df5f250b1508575aae3c9c7cb25964c88"
Oct 30 04:00:22.144369 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:22.144344    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd_odh-ai-gateway-infra(eaf0d3e6-b75e-4652-a4f1-a348c357dc9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e"
Oct 30 04:00:22.147075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.147020    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c" event={"ID":"fe99e906-f412-4f34-91d1-c00607c1f067","Type":"ContainerDied","Data":"069195a4094677124596bde68a02d79e552532e11307d67bfe9e203367cc48f2"}
Oct 30 04:00:22.147186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.147069    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"
Oct 30 04:00:22.150492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.148807    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8" event={"ID":"ad2f1fa7-7466-4640-935e-f78f3aa91ad5","Type":"ContainerDied","Data":"26d5d0d70f08abe4c16fada6d2794615145924f4fded52db84e7e272c90b2c06"}
Oct 30 04:00:22.150492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.148907    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"
Oct 30 04:00:22.158679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.158650    4877 generic.go:356] "Generic (PLEG): container finished" podID="722060d3-d60d-4c96-972d-71e3b1408128" containerID="28637b62a217cc834dc41d1b2f3ae1b95bf6e9ce7ba834212036fa782d5befa7" exitCode=60
Oct 30 04:00:22.158957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.158933    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerDied","Data":"28637b62a217cc834dc41d1b2f3ae1b95bf6e9ce7ba834212036fa782d5befa7"}
Oct 30 04:00:22.159621 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.159600    4877 scope.go:122] "RemoveContainer" containerID="28637b62a217cc834dc41d1b2f3ae1b95bf6e9ce7ba834212036fa782d5befa7"
Oct 30 04:00:22.160925 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:22.160901    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh_odh-ai-gateway-infra(722060d3-d60d-4c96-972d-71e3b1408128)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" podUID="722060d3-d60d-4c96-972d-71e3b1408128"
Oct 30 04:00:22.171501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.170842    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls" event={"ID":"019067eb-aff1-42fe-9c71-7dcf9bcc29b4","Type":"ContainerDied","Data":"dcb845296025b93d8298c178922dc92908673f08d90e7eb55a40ef806758e540"}
Oct 30 04:00:22.171501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.170937    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"
Oct 30 04:00:22.186106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.186045    4877 scope.go:122] "RemoveContainer" containerID="a705269b1a2e2bdc28636c38fb8d474ef2bfa0626443fa644be814a6adbb70df"
Oct 30 04:00:22.190900 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.190873    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clhw8\" (UniqueName: \"kubernetes.io/projected/008f7da2-1318-40d9-b589-bb94835510e6-kube-api-access-clhw8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.195234 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.195206    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt" event={"ID":"989c1932-7a7a-4f64-93f5-6a46bd823d89","Type":"ContainerDied","Data":"0676e712015597058620f789e42a1b6fe126d988a4ff76ccd60b78476f1d6fa6"}
Oct 30 04:00:22.195412 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.195324    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"
Oct 30 04:00:22.208462 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.208192    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n" event={"ID":"836b510f-a531-4569-9bc1-3416e47cafee","Type":"ContainerDied","Data":"81cdd8caa266e4c89ada4d58e6b3c9c6c1fb6c0984041de44e560eadcdce5ef3"}
Oct 30 04:00:22.208462 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.208326    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"
Oct 30 04:00:22.251806 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.251565    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerStarted","Data":"547dabf2a1b5c7644f5aa8ae71df6b73faa190ce9559038f5aca414e1559a914"}
Oct 30 04:00:22.254168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.253309    4877 scope.go:122] "RemoveContainer" containerID="547dabf2a1b5c7644f5aa8ae71df6b73faa190ce9559038f5aca414e1559a914"
Oct 30 04:00:22.284047 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:22.284006    4877 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb_odh-ai-gateway-infra(1e98105c-1514-413e-9bce-2cbc6761294a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" podUID="1e98105c-1514-413e-9bce-2cbc6761294a"
Oct 30 04:00:22.284395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.273947    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"]
Oct 30 04:00:22.284636 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.284616    4877 scope.go:122] "RemoveContainer" containerID="7a2ff496408c09265a13541f34e5d6add5d5e6ca6bfebbe686bed28da0fa98d1"
Oct 30 04:00:22.285960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.285335    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp" event={"ID":"008f7da2-1318-40d9-b589-bb94835510e6","Type":"ContainerDied","Data":"6ac4c1136d64ffcb5b2e0941f313f5f98dca2b147e4c0474b8b45b05503b25b5"}
Oct 30 04:00:22.285960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.285349    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"
Oct 30 04:00:22.311548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.310583    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"]
Oct 30 04:00:22.316023 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.315989    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3d585-29888880-r298c"]
Oct 30 04:00:22.344481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.344374    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"]
Oct 30 04:00:22.349478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.345900    4877 scope.go:122] "RemoveContainer" containerID="48850f624f15b8cb4aa2a17edceae10f390032a91cec9a497a18dc4964fcb45e"
Oct 30 04:00:22.374870 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.374180    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e10c0f-29888880-vh9ls"]
Oct 30 04:00:22.393066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.392977    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"]
Oct 30 04:00:22.426880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.426854    4877 scope.go:122] "RemoveContainer" containerID="71304defb65933cdb9c7f74afe3c9caf397675556563061d8a89dedfec3e8bd6"
Oct 30 04:00:22.441801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.441781    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:22.473157 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.473131    4877 scope.go:122] "RemoveContainer" containerID="605b31839cedf7a4da07e50dd8f7b72bf25d611c299a32036535a09cda0ed5ed"
Oct 30 04:00:22.478264 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.478242    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e1a0c2-29888880-lpsf8"]
Oct 30 04:00:22.496583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.496567    4877 scope.go:122] "RemoveContainer" containerID="97a6498e068dad060d7f68d1297201123deb3d43e431271cfacbd3983e412b63"
Oct 30 04:00:22.499994 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.499967    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"]
Oct 30 04:00:22.541823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.541798    4877 scope.go:122] "RemoveContainer" containerID="bbe8bcf6d8f3a02bbd859793f918c31b597f863933063094162adfd14a7887dc"
Oct 30 04:00:22.545507 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.545482    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:22.551352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.551321    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:22.561763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.561740    4877 scope.go:122] "RemoveContainer" containerID="93f2371d54b412e94dbceb779eeba987189da7d402bd7a5002f718d75aafeb05"
Oct 30 04:00:22.598597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.598567    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca\") pod \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") "
Oct 30 04:00:22.598704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.598634    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl\" (UniqueName: \"kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl\") pod \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\" (UID: \"7ced66d5-2223-4af9-9a6d-9de05d1f6750\") "
Oct 30 04:00:22.598987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.598960    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca" pod "7ced66d5-2223-4af9-9a6d-9de05d1f6750" (UID: "7ced66d5-2223-4af9-9a6d-9de05d1f6750"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.601017 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.600995    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ced66d5-2223-4af9-9a6d-9de05d1f6750-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.601993 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.601969    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl" pod "7ced66d5-2223-4af9-9a6d-9de05d1f6750" (UID: "7ced66d5-2223-4af9-9a6d-9de05d1f6750"). InnerVolumeSpecName "kube-api-access-wc4rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.614649 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.614557    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:22.622599 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.622582    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:22.627845 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.627819    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:22.666368 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.666337    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:22.674883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.674837    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"]
Oct 30 04:00:22.682452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.682410    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e44313-29888880-g4mcp"]
Oct 30 04:00:22.702349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702321    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca\") pod \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") "
Oct 30 04:00:22.702548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702366    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca\") pod \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") "
Oct 30 04:00:22.702548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702474    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg\" (UniqueName: \"kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg\") pod \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") "
Oct 30 04:00:22.702548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702521    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg\" (UniqueName: \"kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg\") pod \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\" (UID: \"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9\") "
Oct 30 04:00:22.702745 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702618    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb\" (UniqueName: \"kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb\") pod \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\" (UID: \"e81dceac-bb98-4f7c-9995-d91a2fef84d0\") "
Oct 30 04:00:22.702745 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702687    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt\" (UniqueName: \"kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt\") pod \"d9413205-622e-4cee-82f9-e6c9fd2041fe\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") "
Oct 30 04:00:22.702745 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702717    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca\") pod \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\" (UID: \"e7a919ba-751f-4778-8d8c-7cd445d1c3d2\") "
Oct 30 04:00:22.702745 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702725    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca" pod "d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" (UID: "d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.703315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702752    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca\") pod \"d9413205-622e-4cee-82f9-e6c9fd2041fe\" (UID: \"d9413205-622e-4cee-82f9-e6c9fd2041fe\") "
Oct 30 04:00:22.703315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702759    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca" pod "e81dceac-bb98-4f7c-9995-d91a2fef84d0" (UID: "e81dceac-bb98-4f7c-9995-d91a2fef84d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.703315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.702782    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8\" (UniqueName: \"kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8\") pod \"e7409641-adac-4021-97f5-b3fcffb172a8\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") "
Oct 30 04:00:22.703315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.703112    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca\") pod \"e7409641-adac-4021-97f5-b3fcffb172a8\" (UID: \"e7409641-adac-4021-97f5-b3fcffb172a8\") "
Oct 30 04:00:22.703315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.703195    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca" pod "e7a919ba-751f-4778-8d8c-7cd445d1c3d2" (UID: "e7a919ba-751f-4778-8d8c-7cd445d1c3d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.703315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.703200    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca" pod "d9413205-622e-4cee-82f9-e6c9fd2041fe" (UID: "d9413205-622e-4cee-82f9-e6c9fd2041fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.703986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.703959    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81dceac-bb98-4f7c-9995-d91a2fef84d0-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.704153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.703993    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.704153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.704009    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4rl\" (UniqueName: \"kubernetes.io/projected/7ced66d5-2223-4af9-9a6d-9de05d1f6750-kube-api-access-wc4rl\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.704153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.704025    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.704153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.704040    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9413205-622e-4cee-82f9-e6c9fd2041fe-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.704471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.704297    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca" pod "e7409641-adac-4021-97f5-b3fcffb172a8" (UID: "e7409641-adac-4021-97f5-b3fcffb172a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.705208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.705180    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg" pod "e7a919ba-751f-4778-8d8c-7cd445d1c3d2" (UID: "e7a919ba-751f-4778-8d8c-7cd445d1c3d2"). InnerVolumeSpecName "kube-api-access-cmxwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.705694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.705664    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg" pod "d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" (UID: "d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9"). InnerVolumeSpecName "kube-api-access-jdfpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.707122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.707090    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8" pod "e7409641-adac-4021-97f5-b3fcffb172a8" (UID: "e7409641-adac-4021-97f5-b3fcffb172a8"). InnerVolumeSpecName "kube-api-access-9vts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.707235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.707140    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt" pod "d9413205-622e-4cee-82f9-e6c9fd2041fe" (UID: "d9413205-622e-4cee-82f9-e6c9fd2041fe"). InnerVolumeSpecName "kube-api-access-26rjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.707369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.707351    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb" pod "e81dceac-bb98-4f7c-9995-d91a2fef84d0" (UID: "e81dceac-bb98-4f7c-9995-d91a2fef84d0"). InnerVolumeSpecName "kube-api-access-sr5bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.715247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.715221    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"]
Oct 30 04:00:22.720699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.720677    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b267f8-29888880-dw94n"]
Oct 30 04:00:22.778895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.778861    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"]
Oct 30 04:00:22.789162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.789134    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888880-qtmpt"]
Oct 30 04:00:22.804790 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.804761    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65\" (UniqueName: \"kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65\") pod \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") "
Oct 30 04:00:22.805106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805083    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca\") pod \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\" (UID: \"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c\") "
Oct 30 04:00:22.805606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805579    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca" pod "5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" (UID: "5331a17c-1fa3-46ae-bba8-ec3d5f94f68c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:22.805929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805909    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7409641-adac-4021-97f5-b3fcffb172a8-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.805992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805933    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxwg\" (UniqueName: \"kubernetes.io/projected/e7a919ba-751f-4778-8d8c-7cd445d1c3d2-kube-api-access-cmxwg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.805992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805946    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdfpg\" (UniqueName: \"kubernetes.io/projected/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9-kube-api-access-jdfpg\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.805992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805960    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr5bb\" (UniqueName: \"kubernetes.io/projected/e81dceac-bb98-4f7c-9995-d91a2fef84d0-kube-api-access-sr5bb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.805992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805974    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.805992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.805990    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rjt\" (UniqueName: \"kubernetes.io/projected/d9413205-622e-4cee-82f9-e6c9fd2041fe-kube-api-access-26rjt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.806234 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.806003    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vts8\" (UniqueName: \"kubernetes.io/projected/e7409641-adac-4021-97f5-b3fcffb172a8-kube-api-access-9vts8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.807248 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.807221    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65" pod "5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" (UID: "5331a17c-1fa3-46ae-bba8-ec3d5f94f68c"). InnerVolumeSpecName "kube-api-access-qmq65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:22.816984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.816953    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"]
Oct 30 04:00:22.907710 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.907677    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmq65\" (UniqueName: \"kubernetes.io/projected/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c-kube-api-access-qmq65\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:22.919563 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:22.919529    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"]
Oct 30 04:00:23.143349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.143262    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"]
Oct 30 04:00:23.143825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.143550    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy" containerID="cri-o://b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056" gracePeriod=30
Oct 30 04:00:23.149789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.148934    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"]
Oct 30 04:00:23.150224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.150195    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy" containerID="cri-o://ef4f5549cde91989d463efc9b556f51661a630d4786afdbb8cf085630895a928" gracePeriod=30
Oct 30 04:00:23.358452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.349598    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" podUID="a24f7bc8-24b0-463b-9667-450610471d42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:00:23.382663 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.382618    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:00:23.421784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.421688    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n" event={"ID":"d9413205-622e-4cee-82f9-e6c9fd2041fe","Type":"ContainerDied","Data":"2256c7f2530292578c866c9172e71c0b14ac94339e8e62f429605646fd6162df"}
Oct 30 04:00:23.435443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.422035    4877 scope.go:122] "RemoveContainer" containerID="d18ef083b529c964386a8ccd12631544be988ce2ab7e2fa5235549f2e7e6bc74"
Oct 30 04:00:23.435443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.422169    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"
Oct 30 04:00:23.539800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.539762    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr" event={"ID":"e7409641-adac-4021-97f5-b3fcffb172a8","Type":"ContainerDied","Data":"bd06cf219d1638d43175e8d5e614d27b027f7bc5612668e0028dc46dd7e725ab"}
Oct 30 04:00:23.539927 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.539870    4877 scope.go:122] "RemoveContainer" containerID="384870fd756d2568d162d02c1f737d7b0060ef250a6320bd0f3578b56e86279c"
Oct 30 04:00:23.539997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.539981    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"
Oct 30 04:00:23.555943 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.555919    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6" event={"ID":"e7a919ba-751f-4778-8d8c-7cd445d1c3d2","Type":"ContainerDied","Data":"a15d52cdb71ef31baa93fb220d61e9196d78b645ee151e4ecf6ab2dcc6233833"}
Oct 30 04:00:23.557799 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.556088    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"
Oct 30 04:00:23.558104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.558087    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"]
Oct 30 04:00:23.570496 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.570444    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc" event={"ID":"e81dceac-bb98-4f7c-9995-d91a2fef84d0","Type":"ContainerDied","Data":"43fd94e694e859e2d82a55e3b321b0cb16d54353a48de14fcdcb2542d637a6ff"}
Oct 30 04:00:23.570811 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.570791    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"
Oct 30 04:00:23.576504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.574441    4877 generic.go:356] "Generic (PLEG): container finished" podID="1e98105c-1514-413e-9bce-2cbc6761294a" containerID="547dabf2a1b5c7644f5aa8ae71df6b73faa190ce9559038f5aca414e1559a914" exitCode=60
Oct 30 04:00:23.576504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.574575    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerDied","Data":"547dabf2a1b5c7644f5aa8ae71df6b73faa190ce9559038f5aca414e1559a914"}
Oct 30 04:00:23.580929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.580899    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m" event={"ID":"d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9","Type":"ContainerDied","Data":"0f25251c1d4d512e2e282c1b25c29f1986ce246420a465c139a1cd6c96a71632"}
Oct 30 04:00:23.581082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.581066    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"
Oct 30 04:00:23.584282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.584251    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8" event={"ID":"5331a17c-1fa3-46ae-bba8-ec3d5f94f68c","Type":"ContainerDied","Data":"f8545c4f984954bf614b2cd1d3a334a8422c5da90b2a97d4ae026c86d901eca1"}
Oct 30 04:00:23.584386 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.584370    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"
Oct 30 04:00:23.589316 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.586637    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms" event={"ID":"7ced66d5-2223-4af9-9a6d-9de05d1f6750","Type":"ContainerDied","Data":"828ce6bbd6735d67312588b4f3cc5e82c90e8a6d78961c2334433b63859b8750"}
Oct 30 04:00:23.589316 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.586740    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"
Oct 30 04:00:23.593884 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.593846    4877 scope.go:122] "RemoveContainer" containerID="2195dc81f77e80fd7da0ae09e3f61f57fc7b71edbbedd57600bd99e324fd8361"
Oct 30 04:00:23.624504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.624475    4877 scope.go:122] "RemoveContainer" containerID="7fe32821d179f607e76146bee8e55110280ab8d010f35d28c601324fbbea4b96"
Oct 30 04:00:23.660946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.660828    4877 scope.go:122] "RemoveContainer" containerID="081a27215c4f6230e6b1b3c35aac0870179721fadadbfd8086938a7ae5936f1f"
Oct 30 04:00:23.663607 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.663552    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"]
Oct 30 04:00:23.675552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.675441    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888880-l58qr"]
Oct 30 04:00:23.691615 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.691585    4877 scope.go:122] "RemoveContainer" containerID="d1c6c16128c1ccfdeca97336fac92cee7b22c2a427197eafd0379542cb287ea1"
Oct 30 04:00:23.716482 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.716453    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008f7da2-1318-40d9-b589-bb94835510e6" path="/var/lib/kubelet/pods/008f7da2-1318-40d9-b589-bb94835510e6/volumes"
Oct 30 04:00:23.717324 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.717303    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" path="/var/lib/kubelet/pods/019067eb-aff1-42fe-9c71-7dcf9bcc29b4/volumes"
Oct 30 04:00:23.718179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.718159    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836b510f-a531-4569-9bc1-3416e47cafee" path="/var/lib/kubelet/pods/836b510f-a531-4569-9bc1-3416e47cafee/volumes"
Oct 30 04:00:23.719305 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.719285    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" path="/var/lib/kubelet/pods/989c1932-7a7a-4f64-93f5-6a46bd823d89/volumes"
Oct 30 04:00:23.720661 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.720641    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" path="/var/lib/kubelet/pods/aa661a42-aaa6-4ff8-947f-3962a969dac8/volumes"
Oct 30 04:00:23.721394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.721373    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" path="/var/lib/kubelet/pods/ad2f1fa7-7466-4640-935e-f78f3aa91ad5/volumes"
Oct 30 04:00:23.728110 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.728089    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" path="/var/lib/kubelet/pods/e7409641-adac-4021-97f5-b3fcffb172a8/volumes"
Oct 30 04:00:23.728861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.728842    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" path="/var/lib/kubelet/pods/fe99e906-f412-4f34-91d1-c00607c1f067/volumes"
Oct 30 04:00:23.731949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.731935    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"]
Oct 30 04:00:23.732135 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.732067    4877 scope.go:122] "RemoveContainer" containerID="94799d88d9646575dbe446532f0a451655498fe4cb87493d7c1e9309966555ca"
Oct 30 04:00:23.739441 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.739406    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888880-cbq2n"]
Oct 30 04:00:23.749639 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.749163    4877 scope.go:122] "RemoveContainer" containerID="6687e9389d224fff2fbccae55ffe23c5267f45f2ef4f7b87a920ded815e130bc"
Oct 30 04:00:23.762511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.762482    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"]
Oct 30 04:00:23.774819 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.773575    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2d5b09-29888880-nl47m"]
Oct 30 04:00:23.802287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.802254    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"]
Oct 30 04:00:23.810974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.810683    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-441461-29888880-vmmc6"]
Oct 30 04:00:23.835491 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.835463    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"]
Oct 30 04:00:23.840578 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.838003    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:23.849232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.849208    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888880-l6hms"]
Oct 30 04:00:23.878548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.878519    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"]
Oct 30 04:00:23.883608 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.883583    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3c874f-29888880-8cdr8"]
Oct 30 04:00:23.908876 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.908849    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"]
Oct 30 04:00:23.913063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.913033    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca\") pod \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") "
Oct 30 04:00:23.913406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.913383    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn\" (UniqueName: \"kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn\") pod \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\" (UID: \"e24f3539-e5a7-4de6-9c42-ca52c8de6344\") "
Oct 30 04:00:23.913769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.913746    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca" pod "e24f3539-e5a7-4de6-9c42-ca52c8de6344" (UID: "e24f3539-e5a7-4de6-9c42-ca52c8de6344"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:23.914273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.914206    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24f3539-e5a7-4de6-9c42-ca52c8de6344-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:23.915786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.915751    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn" pod "e24f3539-e5a7-4de6-9c42-ca52c8de6344" (UID: "e24f3539-e5a7-4de6-9c42-ca52c8de6344"). InnerVolumeSpecName "kube-api-access-6lvhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:23.916097 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.916076    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-547a78-29888880-m2crc"]
Oct 30 04:00:23.937216 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.937192    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:23.940979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.940942    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:23.945281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:23.945262    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:24.014988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.014960    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv\" (UniqueName: \"kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv\") pod \"1e98105c-1514-413e-9bce-2cbc6761294a\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") "
Oct 30 04:00:24.015173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015027    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca\") pod \"1e98105c-1514-413e-9bce-2cbc6761294a\" (UID: \"1e98105c-1514-413e-9bce-2cbc6761294a\") "
Oct 30 04:00:24.015231 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015214    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca\") pod \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") "
Oct 30 04:00:24.015381 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015359    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca\") pod \"722060d3-d60d-4c96-972d-71e3b1408128\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") "
Oct 30 04:00:24.015478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015459    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw\" (UniqueName: \"kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw\") pod \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\" (UID: \"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e\") "
Oct 30 04:00:24.015580 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015564    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp\" (UniqueName: \"kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp\") pod \"722060d3-d60d-4c96-972d-71e3b1408128\" (UID: \"722060d3-d60d-4c96-972d-71e3b1408128\") "
Oct 30 04:00:24.015712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015687    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca" pod "eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" (UID: "eaf0d3e6-b75e-4652-a4f1-a348c357dc9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:24.015777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.015703    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca" pod "1e98105c-1514-413e-9bce-2cbc6761294a" (UID: "1e98105c-1514-413e-9bce-2cbc6761294a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:24.016045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.016023    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca" pod "722060d3-d60d-4c96-972d-71e3b1408128" (UID: "722060d3-d60d-4c96-972d-71e3b1408128"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:24.016490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.016464    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/722060d3-d60d-4c96-972d-71e3b1408128-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.016589 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.016495    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98105c-1514-413e-9bce-2cbc6761294a-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.016589 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.016509    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-maas-api-service-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.016589 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.016523    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvhn\" (UniqueName: \"kubernetes.io/projected/e24f3539-e5a7-4de6-9c42-ca52c8de6344-kube-api-access-6lvhn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.018322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.018287    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp" pod "722060d3-d60d-4c96-972d-71e3b1408128" (UID: "722060d3-d60d-4c96-972d-71e3b1408128"). InnerVolumeSpecName "kube-api-access-vb5jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:24.018598 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.018571    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw" pod "eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" (UID: "eaf0d3e6-b75e-4652-a4f1-a348c357dc9e"). InnerVolumeSpecName "kube-api-access-k4kxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:24.018731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.018646    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv" pod "1e98105c-1514-413e-9bce-2cbc6761294a" (UID: "1e98105c-1514-413e-9bce-2cbc6761294a"). InnerVolumeSpecName "kube-api-access-qpqtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:24.119815 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.119772    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4kxw\" (UniqueName: \"kubernetes.io/projected/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e-kube-api-access-k4kxw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.119815 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.119816    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5jp\" (UniqueName: \"kubernetes.io/projected/722060d3-d60d-4c96-972d-71e3b1408128-kube-api-access-vb5jp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.120263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.119832    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqtv\" (UniqueName: \"kubernetes.io/projected/1e98105c-1514-413e-9bce-2cbc6761294a-kube-api-access-qpqtv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:24.603207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.603170    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"
Oct 30 04:00:24.603207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.603184    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk" event={"ID":"e24f3539-e5a7-4de6-9c42-ca52c8de6344","Type":"ContainerDied","Data":"381aaa31d64b1d0a654799c59db0bbe7718d7f9b629c47e7d4f515524a753556"}
Oct 30 04:00:24.603842 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.603288    4877 scope.go:122] "RemoveContainer" containerID="fd44839b463d2c485fde34799f38a77179da879acc7ce44df584b0f91eb6f848"
Oct 30 04:00:24.605180 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.604917    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd" event={"ID":"eaf0d3e6-b75e-4652-a4f1-a348c357dc9e","Type":"ContainerDied","Data":"5d0da999ea73a7c939fa06e5715892050da31b787e62bc61db094b0d275e4fed"}
Oct 30 04:00:24.605180 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.605049    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"
Oct 30 04:00:24.610883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.610858    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"
Oct 30 04:00:24.610883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.610872    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh" event={"ID":"722060d3-d60d-4c96-972d-71e3b1408128","Type":"ContainerDied","Data":"0a4b9c0f6937a3d373f30ed43d7975bc2b8b6763911a66a3cdcab53df17caef3"}
Oct 30 04:00:24.617600 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.617581    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"
Oct 30 04:00:24.617778 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.617637    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb" event={"ID":"1e98105c-1514-413e-9bce-2cbc6761294a","Type":"ContainerDied","Data":"ac804718e9794e894cb018d863385fb795e82e844edc1c6ff2b10b932dfa5faa"}
Oct 30 04:00:24.619680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.619573    4877 scope.go:122] "RemoveContainer" containerID="a30c9392c185e3e345de90002d15687df5f250b1508575aae3c9c7cb25964c88"
Oct 30 04:00:24.634566 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.634543    4877 scope.go:122] "RemoveContainer" containerID="28637b62a217cc834dc41d1b2f3ae1b95bf6e9ce7ba834212036fa782d5befa7"
Oct 30 04:00:24.645003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.644983    4877 scope.go:122] "RemoveContainer" containerID="547dabf2a1b5c7644f5aa8ae71df6b73faa190ce9559038f5aca414e1559a914"
Oct 30 04:00:24.674968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.674945    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"]
Oct 30 04:00:24.682041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.682014    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-1a1b45-29888880-z4jjd"]
Oct 30 04:00:24.709181 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.709153    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"]
Oct 30 04:00:24.714219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.714196    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-855e14-29888880-sxpvh"]
Oct 30 04:00:24.743339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.743317    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"]
Oct 30 04:00:24.748196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.748174    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-da4553-29888880-tswsb"]
Oct 30 04:00:24.801601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.801551    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"]
Oct 30 04:00:24.813187 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:24.813151    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-372480-29888880-xcbzk"]
Oct 30 04:00:25.702312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.702275    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" path="/var/lib/kubelet/pods/1e98105c-1514-413e-9bce-2cbc6761294a/volumes"
Oct 30 04:00:25.702993 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.702974    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" path="/var/lib/kubelet/pods/5331a17c-1fa3-46ae-bba8-ec3d5f94f68c/volumes"
Oct 30 04:00:25.703452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.703406    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722060d3-d60d-4c96-972d-71e3b1408128" path="/var/lib/kubelet/pods/722060d3-d60d-4c96-972d-71e3b1408128/volumes"
Oct 30 04:00:25.704196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.704177    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" path="/var/lib/kubelet/pods/7ced66d5-2223-4af9-9a6d-9de05d1f6750/volumes"
Oct 30 04:00:25.704675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.704649    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" path="/var/lib/kubelet/pods/d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9/volumes"
Oct 30 04:00:25.705075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.705060    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" path="/var/lib/kubelet/pods/d9413205-622e-4cee-82f9-e6c9fd2041fe/volumes"
Oct 30 04:00:25.705781 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.705754    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" path="/var/lib/kubelet/pods/e24f3539-e5a7-4de6-9c42-ca52c8de6344/volumes"
Oct 30 04:00:25.706255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.706234    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" path="/var/lib/kubelet/pods/e7a919ba-751f-4778-8d8c-7cd445d1c3d2/volumes"
Oct 30 04:00:25.706764 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.706742    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" path="/var/lib/kubelet/pods/e81dceac-bb98-4f7c-9995-d91a2fef84d0/volumes"
Oct 30 04:00:25.707529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:25.707509    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" path="/var/lib/kubelet/pods/eaf0d3e6-b75e-4652-a4f1-a348c357dc9e/volumes"
Oct 30 04:00:27.077702 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.077666    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"]
Oct 30 04:00:27.078464 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078442    4877 cpu_manager.go:414] "containerMap: removing container" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.078464 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078458    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078471    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078476    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078485    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078491    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078498    4877 cpu_manager.go:414] "containerMap: removing container" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078503    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078511    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e5316ebf-1c98-4b55-af72-243224166688" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078516    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5316ebf-1c98-4b55-af72-243224166688" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078523    4877 cpu_manager.go:414] "containerMap: removing container" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078527    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078535    4877 cpu_manager.go:414] "containerMap: removing container" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078539    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078545    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078549    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078558    4877 cpu_manager.go:414] "containerMap: removing container" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078562    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.078571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078575    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078580    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078586    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078591    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078599    4877 cpu_manager.go:414] "containerMap: removing container" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078603    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078614    4877 cpu_manager.go:414] "containerMap: removing container" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078620    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078628    4877 cpu_manager.go:414] "containerMap: removing container" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078632    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078641    4877 cpu_manager.go:414] "containerMap: removing container" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078645    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078651    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078655    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078662    4877 cpu_manager.go:414] "containerMap: removing container" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078666    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078673    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078678    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078690    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078694    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078702    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078706    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078714    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078718    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078725    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078729    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078739    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078744    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078754    4877 cpu_manager.go:414] "containerMap: removing container" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078758    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078765    4877 cpu_manager.go:414] "containerMap: removing container" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078770    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078777    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078782    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078790    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078794    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078800    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078805    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078813    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078817    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078824    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078829    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078841    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078846    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078854    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078858    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078865    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078870    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078876    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078880    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078887    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078891    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078898    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078902    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078907    4877 cpu_manager.go:414] "containerMap: removing container" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078912    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078920    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078924    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078931    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078935    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078943    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078947    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078955    4877 cpu_manager.go:414] "containerMap: removing container" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078959    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078966    4877 cpu_manager.go:414] "containerMap: removing container" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078973    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078980    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078986    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.078995    4877 cpu_manager.go:414] "containerMap: removing container" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079001    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079010    4877 cpu_manager.go:414] "containerMap: removing container" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079015    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079028    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079032    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079041    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079045    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079051    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079056    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079061    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.079084 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079066    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079186    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079196    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079201    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079208    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079212    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079218    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079224    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079229    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079234    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079239    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079246    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079253    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079260    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5316ebf-1c98-4b55-af72-243224166688" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079265    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079270    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079276    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079282    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079288    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079295    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5331a17c-1fa3-46ae-bba8-ec3d5f94f68c" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079300    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079306    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079310    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079315    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079322    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079327    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079333    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079338    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24f3539-e5a7-4de6-9c42-ca52c8de6344" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079345    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079350    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079356    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079361    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079367    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079371    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079376    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f1fa7-7466-4640-935e-f78f3aa91ad5" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079382    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079388    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e98105c-1514-413e-9bce-2cbc6761294a" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079394    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079400    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079404    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079411    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079417    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="019067eb-aff1-42fe-9c71-7dcf9bcc29b4" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079434    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079440    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079446    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079452    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079457    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079463    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079469    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="722060d3-d60d-4c96-972d-71e3b1408128" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079475    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079482    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:27.081651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.079488    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:27.085052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.085034    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.099810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.099784    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"]
Oct 30 04:00:27.104073 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.104040    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:00:27.104186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.104088    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:00:27.167157 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.167125    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfhj\" (UniqueName: \"kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.167315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.167181    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.167379 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.167359    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.268921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.268886    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.269120 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.269101    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfhj\" (UniqueName: \"kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.269186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.269171    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.269823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.269797    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.271281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.271264    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.285882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.285857    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfhj\" (UniqueName: \"kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj\") pod \"maas-api-revoke-sub-0012b598-hcg5v\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.401644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.401565    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:27.715565 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:27.715539    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"]
Oct 30 04:00:27.718332 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:27.718301    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75aefa7e_f7a5_485d_b74d_57d50f109b66.slice/crio-126c1597e663f03e08f76d09b5f1eb76723ceb0cf5658b05295a0eddaef9dd43 WatchSource:0}: Error finding container 126c1597e663f03e08f76d09b5f1eb76723ceb0cf5658b05295a0eddaef9dd43: Status 404 returned error can't find the container with id 126c1597e663f03e08f76d09b5f1eb76723ceb0cf5658b05295a0eddaef9dd43
Oct 30 04:00:28.514006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.513979    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 04:00:28.606406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606357    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.606648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606491    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.606742 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606724    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.606843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606827    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606889    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606923    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606954    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.606998    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.607095    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket\") pod \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\" (UID: \"a633a9a2-e8ac-415b-95c3-4d478846e0ce\") "
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.608129    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.609633    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.609779    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.610038    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.610506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.610202    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.614733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.614685    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "kube-api-access-gnhng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:28.615616 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.615587    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.616521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.616482    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:28.620047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.620016    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo" pod "a633a9a2-e8ac-415b-95c3-4d478846e0ce" (UID: "a633a9a2-e8ac-415b-95c3-4d478846e0ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:00:28.663853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.663813    4877 generic.go:356] "Generic (PLEG): container finished" podID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerID="b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056" exitCode=0
Oct 30 04:00:28.663999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.663866    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"
Oct 30 04:00:28.663999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.663894    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" event={"ID":"a633a9a2-e8ac-415b-95c3-4d478846e0ce","Type":"ContainerDied","Data":"b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056"}
Oct 30 04:00:28.663999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.663965    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7" event={"ID":"a633a9a2-e8ac-415b-95c3-4d478846e0ce","Type":"ContainerDied","Data":"2c6c451808aa243a5083fd4a300067495fad6b1dbee7f1cdd80e7bf89300214e"}
Oct 30 04:00:28.663999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.663995    4877 scope.go:122] "RemoveContainer" containerID="b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056"
Oct 30 04:00:28.665868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.665845    4877 generic.go:356] "Generic (PLEG): container finished" podID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerID="ef4f5549cde91989d463efc9b556f51661a630d4786afdbb8cf085630895a928" exitCode=0
Oct 30 04:00:28.666015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.665916    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" event={"ID":"73a85e54-af6e-4b2f-8619-a69e45d9da7c","Type":"ContainerDied","Data":"ef4f5549cde91989d463efc9b556f51661a630d4786afdbb8cf085630895a928"}
Oct 30 04:00:28.666015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.665958    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw" event={"ID":"73a85e54-af6e-4b2f-8619-a69e45d9da7c","Type":"ContainerDied","Data":"0a358f139f1b1ba99e22eebdbfb976b22ef13d5e78a018137052b394ca561c3e"}
Oct 30 04:00:28.666015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.665982    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a358f139f1b1ba99e22eebdbfb976b22ef13d5e78a018137052b394ca561c3e"
Oct 30 04:00:28.667910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.667874    4877 generic.go:356] "Generic (PLEG): container finished" podID="75aefa7e-f7a5-485d-b74d-57d50f109b66" containerID="957592513e890b53437f2f84e7994c3b556b46bcaf756001b3c8ae38b0128c6f" exitCode=0
Oct 30 04:00:28.668023 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.667987    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v" event={"ID":"75aefa7e-f7a5-485d-b74d-57d50f109b66","Type":"ContainerDied","Data":"957592513e890b53437f2f84e7994c3b556b46bcaf756001b3c8ae38b0128c6f"}
Oct 30 04:00:28.668072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.668028    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v" event={"ID":"75aefa7e-f7a5-485d-b74d-57d50f109b66","Type":"ContainerStarted","Data":"126c1597e663f03e08f76d09b5f1eb76723ceb0cf5658b05295a0eddaef9dd43"}
Oct 30 04:00:28.679236 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.679207    4877 scope.go:122] "RemoveContainer" containerID="d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"
Oct 30 04:00:28.683436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.683404    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 04:00:28.691184 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.691166    4877 scope.go:122] "RemoveContainer" containerID="b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056"
Oct 30 04:00:28.691484 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:28.691461    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056\": container with ID starting with b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056 not found: ID does not exist" containerID="b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056"
Oct 30 04:00:28.691543 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.691493    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056"} err="failed to get container status \"b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056\": rpc error: code = NotFound desc = could not find container \"b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056\": container with ID starting with b7d23f2230fe7b79d507380a3d2644821416c8fe03cee33ebc993d0a40ace056 not found: ID does not exist"
Oct 30 04:00:28.691543 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.691512    4877 scope.go:122] "RemoveContainer" containerID="d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"
Oct 30 04:00:28.691772 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:28.691748    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4\": container with ID starting with d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4 not found: ID does not exist" containerID="d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"
Oct 30 04:00:28.691839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.691782    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4"} err="failed to get container status \"d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4\": rpc error: code = NotFound desc = could not find container \"d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4\": container with ID starting with d65493457400abb2397178f167991873995a5c4a45553c8f39e336e968809ed4 not found: ID does not exist"
Oct 30 04:00:28.709055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709022    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709050    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709062    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709074    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709086    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709095    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709105    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a633a9a2-e8ac-415b-95c3-4d478846e0ce-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709112    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a633a9a2-e8ac-415b-95c3-4d478846e0ce-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.709177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.709121    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhng\" (UniqueName: \"kubernetes.io/projected/a633a9a2-e8ac-415b-95c3-4d478846e0ce-kube-api-access-gnhng\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.782323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.782253    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"]
Oct 30 04:00:28.790119 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.790089    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-tcpr7"]
Oct 30 04:00:28.810698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.810667    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.810834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.810792    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.810910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.810839    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.810910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.810872    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.811059 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811029    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.811124 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811107    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.811197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811168    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.811263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811202    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.811263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811204    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.811378 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811259    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.811378 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811294    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert\") pod \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\" (UID: \"73a85e54-af6e-4b2f-8619-a69e45d9da7c\") "
Oct 30 04:00:28.811378 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811327    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.811576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811388    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.811773 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811748    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:28.811986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811966    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.812078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.811991    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.812078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.812004    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.812078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.812031    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.812078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.812045    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.813697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.813678    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "kube-api-access-2qhvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:28.813756 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.813689    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:28.813892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.813868    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:28.813892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.813873    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo" pod "73a85e54-af6e-4b2f-8619-a69e45d9da7c" (UID: "73a85e54-af6e-4b2f-8619-a69e45d9da7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:00:28.913501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.913401    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qhvq\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-kube-api-access-2qhvq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.913501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.913440    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.913501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.913454    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:28.913501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:28.913467    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a85e54-af6e-4b2f-8619-a69e45d9da7c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:29.680179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:29.680104    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"
Oct 30 04:00:29.701187 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:29.701157    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" path="/var/lib/kubelet/pods/a633a9a2-e8ac-415b-95c3-4d478846e0ce/volumes"
Oct 30 04:00:29.741765 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:29.741732    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"]
Oct 30 04:00:29.759234 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:29.752003    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-4h4mw"]
Oct 30 04:00:29.883202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:29.883169    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:30.029681 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.029648    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca\") pod \"75aefa7e-f7a5-485d-b74d-57d50f109b66\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") "
Oct 30 04:00:30.029883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.029787    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca\") pod \"75aefa7e-f7a5-485d-b74d-57d50f109b66\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") "
Oct 30 04:00:30.029883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.029853    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj\" (UniqueName: \"kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj\") pod \"75aefa7e-f7a5-485d-b74d-57d50f109b66\" (UID: \"75aefa7e-f7a5-485d-b74d-57d50f109b66\") "
Oct 30 04:00:30.030212 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.030181    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca" pod "75aefa7e-f7a5-485d-b74d-57d50f109b66" (UID: "75aefa7e-f7a5-485d-b74d-57d50f109b66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:30.030730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.030700    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.032121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.032089    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca" pod "75aefa7e-f7a5-485d-b74d-57d50f109b66" (UID: "75aefa7e-f7a5-485d-b74d-57d50f109b66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:30.032478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.032415    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj" pod "75aefa7e-f7a5-485d-b74d-57d50f109b66" (UID: "75aefa7e-f7a5-485d-b74d-57d50f109b66"). InnerVolumeSpecName "kube-api-access-pdfhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:30.131984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.131943    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdfhj\" (UniqueName: \"kubernetes.io/projected/75aefa7e-f7a5-485d-b74d-57d50f109b66-kube-api-access-pdfhj\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.131984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.131976    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75aefa7e-f7a5-485d-b74d-57d50f109b66-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:30.690739 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.690707    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"
Oct 30 04:00:30.691255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.690741    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v" event={"ID":"75aefa7e-f7a5-485d-b74d-57d50f109b66","Type":"ContainerDied","Data":"126c1597e663f03e08f76d09b5f1eb76723ceb0cf5658b05295a0eddaef9dd43"}
Oct 30 04:00:30.691332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:30.691287    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126c1597e663f03e08f76d09b5f1eb76723ceb0cf5658b05295a0eddaef9dd43"
Oct 30 04:00:31.704944 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:31.704909    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" path="/var/lib/kubelet/pods/73a85e54-af6e-4b2f-8619-a69e45d9da7c/volumes"
Oct 30 04:00:33.028564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:33.028528    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"]
Oct 30 04:00:33.033025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:33.032998    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318ae60-v79dc"]
Oct 30 04:00:33.284374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:33.284275    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" podUID="a24f7bc8-24b0-463b-9667-450610471d42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:00:33.284579 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:33.284477    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:00:33.704764 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:33.704651    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990e7fc4-0c89-45f9-88fb-a118b6df9796" path="/var/lib/kubelet/pods/990e7fc4-0c89-45f9-88fb-a118b6df9796/volumes"
Oct 30 04:00:37.739223 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:37.739183    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"]
Oct 30 04:00:37.739726 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:37.739482    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy" containerID="cri-o://97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c" gracePeriod=30
Oct 30 04:00:38.398720 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.398692    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 04:00:38.439202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.439176    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 04:00:38.479980 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.479937    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume\") pod \"a24f7bc8-24b0-463b-9667-450610471d42\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") "
Oct 30 04:00:38.480204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.480184    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2\" (UniqueName: \"kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2\") pod \"a24f7bc8-24b0-463b-9667-450610471d42\" (UID: \"a24f7bc8-24b0-463b-9667-450610471d42\") "
Oct 30 04:00:38.482760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.482413    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume" pod "a24f7bc8-24b0-463b-9667-450610471d42" (UID: "a24f7bc8-24b0-463b-9667-450610471d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:38.488217 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.488180    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2" pod "a24f7bc8-24b0-463b-9667-450610471d42" (UID: "a24f7bc8-24b0-463b-9667-450610471d42"). InnerVolumeSpecName "kube-api-access-zkmv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:38.581968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.581877    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume\") pod \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") "
Oct 30 04:00:38.582165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.582021    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf\" (UniqueName: \"kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf\") pod \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\" (UID: \"28cf83a4-3795-43c8-86b7-3c5a4c9a2090\") "
Oct 30 04:00:38.582273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.582249    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume" pod "28cf83a4-3795-43c8-86b7-3c5a4c9a2090" (UID: "28cf83a4-3795-43c8-86b7-3c5a4c9a2090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:38.582750 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.582727    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkmv2\" (UniqueName: \"kubernetes.io/projected/a24f7bc8-24b0-463b-9667-450610471d42-kube-api-access-zkmv2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:38.582750 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.582750    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f7bc8-24b0-463b-9667-450610471d42-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:38.582913 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.582760    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:38.584173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.584151    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf" pod "28cf83a4-3795-43c8-86b7-3c5a4c9a2090" (UID: "28cf83a4-3795-43c8-86b7-3c5a4c9a2090"). InnerVolumeSpecName "kube-api-access-896jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:38.684223 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.684189    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896jf\" (UniqueName: \"kubernetes.io/projected/28cf83a4-3795-43c8-86b7-3c5a4c9a2090-kube-api-access-896jf\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:38.772619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.772582    4877 generic.go:356] "Generic (PLEG): container finished" podID="a24f7bc8-24b0-463b-9667-450610471d42" containerID="ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065" exitCode=0
Oct 30 04:00:38.773070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.772654    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"
Oct 30 04:00:38.773070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.772671    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" event={"ID":"a24f7bc8-24b0-463b-9667-450610471d42","Type":"ContainerDied","Data":"ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065"}
Oct 30 04:00:38.773070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.772739    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82" event={"ID":"a24f7bc8-24b0-463b-9667-450610471d42","Type":"ContainerDied","Data":"8ea10bcdc1d25d1ae6e0f46e7f1485c06e9fbaa6b82c8523cc05c2c1e157a04e"}
Oct 30 04:00:38.773070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.772770    4877 scope.go:122] "RemoveContainer" containerID="ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065"
Oct 30 04:00:38.774522 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.774494    4877 generic.go:356] "Generic (PLEG): container finished" podID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerID="ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279" exitCode=0
Oct 30 04:00:38.774630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.774535    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" event={"ID":"28cf83a4-3795-43c8-86b7-3c5a4c9a2090","Type":"ContainerDied","Data":"ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279"}
Oct 30 04:00:38.774630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.774575    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"
Oct 30 04:00:38.774630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.774593    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz" event={"ID":"28cf83a4-3795-43c8-86b7-3c5a4c9a2090","Type":"ContainerDied","Data":"bdb0a41b7d9ee06673f20f73d6b0ff2cd7e2f46ce8fdfc4e55ebef582983632f"}
Oct 30 04:00:38.787815 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.787791    4877 scope.go:122] "RemoveContainer" containerID="ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065"
Oct 30 04:00:38.788147 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:38.788122    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065\": container with ID starting with ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065 not found: ID does not exist" containerID="ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065"
Oct 30 04:00:38.788195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.788159    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065"} err="failed to get container status \"ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065\": rpc error: code = NotFound desc = could not find container \"ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065\": container with ID starting with ed800d896d8ac15608168f71f678b6c6bad82a2a99c5d3d173b03aa9110aa065 not found: ID does not exist"
Oct 30 04:00:38.788195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.788182    4877 scope.go:122] "RemoveContainer" containerID="ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279"
Oct 30 04:00:38.801989 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.801966    4877 scope.go:122] "RemoveContainer" containerID="ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279"
Oct 30 04:00:38.802351 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:38.802327    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279\": container with ID starting with ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279 not found: ID does not exist" containerID="ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279"
Oct 30 04:00:38.802462 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.802360    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279"} err="failed to get container status \"ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279\": rpc error: code = NotFound desc = could not find container \"ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279\": container with ID starting with ff20a366180b8f8a995f5ded30971b6c7ae2eb50495b0b7b09ce636b00714279 not found: ID does not exist"
Oct 30 04:00:38.820471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.820402    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"]
Oct 30 04:00:38.826479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.826454    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a3d585-7965b5f54c-tjfrz"]
Oct 30 04:00:38.851142 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.851078    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"]
Oct 30 04:00:38.856340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:38.856316    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a3d585-5c77ff59cb-gwv82"]
Oct 30 04:00:39.702875 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:39.702836    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" path="/var/lib/kubelet/pods/28cf83a4-3795-43c8-86b7-3c5a4c9a2090/volumes"
Oct 30 04:00:39.703387 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:39.703363    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24f7bc8-24b0-463b-9667-450610471d42" path="/var/lib/kubelet/pods/a24f7bc8-24b0-463b-9667-450610471d42/volumes"
Oct 30 04:00:43.471917 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.471506    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 04:00:43.681498 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.681382    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.683585 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.683556    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.684163 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.684140    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.684620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.684602    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.685032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.685014    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.685523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.685502    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.686103 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.686083    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.686682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.686660    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.687275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.683997    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:43.687275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.684531    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:43.687275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.684945    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:43.687275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.685974    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:43.687275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.686530    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:43.687708 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.687685    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo\") pod \"d276f194-aaa3-4df4-8104-e8844bb5aa82\" (UID: \"d276f194-aaa3-4df4-8104-e8844bb5aa82\") "
Oct 30 04:00:43.688987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.688963    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.689120 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.689104    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.689244 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.689231    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.689339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.689327    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.689529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.689513    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d276f194-aaa3-4df4-8104-e8844bb5aa82-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.726468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.726383    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "kube-api-access-tlc7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:43.727091 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.727046    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:00:43.727330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.727295    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:43.727846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.727807    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token" pod "d276f194-aaa3-4df4-8104-e8844bb5aa82" (UID: "d276f194-aaa3-4df4-8104-e8844bb5aa82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:43.799966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.799880    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.801279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.801253    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlc7g\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-kube-api-access-tlc7g\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.801407 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.801283    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.801407 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.801299    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d276f194-aaa3-4df4-8104-e8844bb5aa82-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:00:43.869623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.863033    4877 generic.go:356] "Generic (PLEG): container finished" podID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerID="97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c" exitCode=0
Oct 30 04:00:43.869623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.863442    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" event={"ID":"d276f194-aaa3-4df4-8104-e8844bb5aa82","Type":"ContainerDied","Data":"97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c"}
Oct 30 04:00:43.869623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.863521    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5" event={"ID":"d276f194-aaa3-4df4-8104-e8844bb5aa82","Type":"ContainerDied","Data":"4246f80468656b2ff8c1cb66d87501e796a320b30aa3c06696af38cc2c434246"}
Oct 30 04:00:43.869623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.863562    4877 scope.go:122] "RemoveContainer" containerID="97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c"
Oct 30 04:00:43.869623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.863728    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"
Oct 30 04:00:43.983413 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.983378    4877 scope.go:122] "RemoveContainer" containerID="fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"
Oct 30 04:00:43.992615 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:43.992594    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"]
Oct 30 04:00:44.108445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:44.082679    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a3d585-openshift-default-85d97fc75f-tnkp5"]
Oct 30 04:00:44.189446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:44.173584    4877 scope.go:122] "RemoveContainer" containerID="97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c"
Oct 30 04:00:44.198634 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:44.198596    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c\": container with ID starting with 97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c not found: ID does not exist" containerID="97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c"
Oct 30 04:00:44.199348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:44.199313    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c"} err="failed to get container status \"97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c\": rpc error: code = NotFound desc = could not find container \"97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c\": container with ID starting with 97e2153da40ebc663473fd6be23c23a4183db930b7d73f67ccf803d3081ce81c not found: ID does not exist"
Oct 30 04:00:44.199523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:44.199505    4877 scope.go:122] "RemoveContainer" containerID="fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"
Oct 30 04:00:44.203442 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:44.200738    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd\": container with ID starting with fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd not found: ID does not exist" containerID="fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"
Oct 30 04:00:44.203442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:44.200777    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd"} err="failed to get container status \"fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd\": rpc error: code = NotFound desc = could not find container \"fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd\": container with ID starting with fa0e156755821ba9dbe2b5217272e45c66103f2b605928df31bc841ee7bab6bd not found: ID does not exist"
Oct 30 04:00:45.705018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:45.704973    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" path="/var/lib/kubelet/pods/d276f194-aaa3-4df4-8104-e8844bb5aa82/volumes"
Oct 30 04:00:46.230648 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:00:46.230615    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:00:50.593700 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.593661    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"]
Oct 30 04:00:50.595109 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595083    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:50.595305 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595282    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595318    4877 cpu_manager.go:414] "containerMap: removing container" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595328    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf0d3e6-b75e-4652-a4f1-a348c357dc9e" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595339    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e5316ebf-1c98-4b55-af72-243224166688" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595346    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5316ebf-1c98-4b55-af72-243224166688" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595359    4877 cpu_manager.go:414] "containerMap: removing container" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595367    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008f7da2-1318-40d9-b589-bb94835510e6" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595386    4877 cpu_manager.go:414] "containerMap: removing container" podUID="75aefa7e-f7a5-485d-b74d-57d50f109b66" containerName="revoke-keys"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595393    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75aefa7e-f7a5-485d-b74d-57d50f109b66" containerName="revoke-keys"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595404    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595411    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ced66d5-2223-4af9-9a6d-9de05d1f6750" containerName="cleanup"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595442    4877 cpu_manager.go:414] "containerMap: removing container" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy"
Oct 30 04:00:50.595451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595451    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595461    4877 cpu_manager.go:414] "containerMap: removing container" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595468    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836b510f-a531-4569-9bc1-3416e47cafee" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595484    4877 cpu_manager.go:414] "containerMap: removing container" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595491    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989c1932-7a7a-4f64-93f5-6a46bd823d89" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595512    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595519    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595531    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595538    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595552    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a24f7bc8-24b0-463b-9667-450610471d42" containerName="payload-processing"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595560    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24f7bc8-24b0-463b-9667-450610471d42" containerName="payload-processing"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595590    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595598    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595607    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595614    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595628    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595635    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595663    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595673    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7409641-adac-4021-97f5-b3fcffb172a8" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595686    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595694    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595718    4877 cpu_manager.go:414] "containerMap: removing container" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerName="payload-pre-processing"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595725    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerName="payload-pre-processing"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595934    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595949    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d276f194-aaa3-4df4-8104-e8844bb5aa82" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595963    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24f7bc8-24b0-463b-9667-450610471d42" containerName="payload-processing"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595977    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c021a4-3f0f-4c64-878a-d9e9cd4d4efd" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.595993    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa661a42-aaa6-4ff8-947f-3962a969dac8" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596004    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c8e20a-d1f8-4c0f-bf6f-fff8aa2c18f9" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596012    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596024    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a633a9a2-e8ac-415b-95c3-4d478846e0ce" containerName="istio-proxy"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596036    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe99e906-f412-4f34-91d1-c00607c1f067" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596048    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5316ebf-1c98-4b55-af72-243224166688" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596061    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81dceac-bb98-4f7c-9995-d91a2fef84d0" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596073    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="28cf83a4-3795-43c8-86b7-3c5a4c9a2090" containerName="payload-pre-processing"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596084    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a919ba-751f-4778-8d8c-7cd445d1c3d2" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596096    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9413205-622e-4cee-82f9-e6c9fd2041fe" containerName="cleanup"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596107    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="75aefa7e-f7a5-485d-b74d-57d50f109b66" containerName="revoke-keys"
Oct 30 04:00:50.596145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.596115    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a85e54-af6e-4b2f-8619-a69e45d9da7c" containerName="istio-proxy"
Oct 30 04:00:50.602749 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.602727    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.620502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.620474    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1597f8bc-openshift-default-dockercfg-4jcpf\""
Oct 30 04:00:50.636395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.636367    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"]
Oct 30 04:00:50.681883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.681826    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682211 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682179    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682232    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682270    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94v5d\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682351    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682450    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682565    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682774    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.682870 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.682837    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799405    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799666 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799497    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799666 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799538    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94v5d\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799666 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799633    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799719    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799789    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.799874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799867    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.800046 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.799923    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.800106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.800081    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.800977 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.800936    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.807517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.801052    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.807726 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.801849    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.807844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.802096    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.807946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.802338    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.824137 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.824099    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.846498 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.831698    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.846727 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.836100    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.862447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.860272    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94v5d\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d\") pod \"e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") " pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:50.939974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:50.938519    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:51.066466 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.066134    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"]
Oct 30 04:00:51.076479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.075529    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11c379d9-hqjrw"]
Oct 30 04:00:51.390305 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:00:51.390274    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab551f7_2d17_4d22_bc7c_cf694c0089ca.slice/crio-4953798ee2aa7192d3aa3ed1d468ec8cda31ac6a7cd54273bc27791ebb957fb2 WatchSource:0}: Error finding container 4953798ee2aa7192d3aa3ed1d468ec8cda31ac6a7cd54273bc27791ebb957fb2: Status 404 returned error can't find the container with id 4953798ee2aa7192d3aa3ed1d468ec8cda31ac6a7cd54273bc27791ebb957fb2
Oct 30 04:00:51.392282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.392215    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"]
Oct 30 04:00:51.393774 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.393741    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:00:51.393866 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.393815    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:00:51.393866 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.393848    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:00:51.707323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:51.704510    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9" path="/var/lib/kubelet/pods/d33200e7-c8e2-4b16-a01e-e0db2fb4ddc9/volumes"
Oct 30 04:00:52.073377 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:52.073338    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" event={"ID":"bab551f7-2d17-4d22-bc7c-cf694c0089ca","Type":"ContainerStarted","Data":"9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"}
Oct 30 04:00:52.073580 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:52.073411    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" event={"ID":"bab551f7-2d17-4d22-bc7c-cf694c0089ca","Type":"ContainerStarted","Data":"4953798ee2aa7192d3aa3ed1d468ec8cda31ac6a7cd54273bc27791ebb957fb2"}
Oct 30 04:00:52.121339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:52.121249    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podStartSLOduration=2.121235281 podStartE2EDuration="2.121235281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:52.119711273 +0000 UTC m=+3480.548600702" watchObservedRunningTime="2026-10-30 04:00:52.121235281 +0000 UTC m=+3480.550124714"
Oct 30 04:00:52.940591 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:52.940551    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:52.946764 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:52.946728    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:53.083238 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:53.083204    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:53.084256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:53.084232    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:00:56.528564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:56.528528    4877 scope.go:122] "RemoveContainer" containerID="0cf554f7f48a9fda54b22251074b869fcd484c17a81f8a00572611066d20e905"
Oct 30 04:00:56.725998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:56.725970    4877 scope.go:122] "RemoveContainer" containerID="b258dcbd506fc4328a6002557a359e1e4198e5afe3365458c0924eb77fc6e040"
Oct 30 04:00:56.739932 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:00:56.739897    4877 scope.go:122] "RemoveContainer" containerID="ef4f5549cde91989d463efc9b556f51661a630d4786afdbb8cf085630895a928"
Oct 30 04:01:05.737472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.737414    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"]
Oct 30 04:01:05.747476 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.747447    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.754775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.754750    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"]
Oct 30 04:01:05.760986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.760955    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:01:05.770806 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.770786    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:01:05.807502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.807407    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.807671 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.807637    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.807742 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.807699    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2mm\" (UniqueName: \"kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.909164 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.909126    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.909349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.909216    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.909349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.909251    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2mm\" (UniqueName: \"kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.909968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.909941    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.911886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.911857    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:05.926004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:05.925979    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2mm\" (UniqueName: \"kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm\") pod \"maas-api-revoke-sub-f0d88702-zrhfq\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:06.063968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:06.063932    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:06.377620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:06.377585    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"]
Oct 30 04:01:06.379936 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:01:06.379564    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78b5171_a071_456e_aac4_5834e5b021f3.slice/crio-f5c8956e259434d26e80ec307f99dd58e292fd3909f5c67c7ee4d0e110c86856 WatchSource:0}: Error finding container f5c8956e259434d26e80ec307f99dd58e292fd3909f5c67c7ee4d0e110c86856: Status 404 returned error can't find the container with id f5c8956e259434d26e80ec307f99dd58e292fd3909f5c67c7ee4d0e110c86856
Oct 30 04:01:07.261865 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:07.261821    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq" event={"ID":"c78b5171-a071-456e-aac4-5834e5b021f3","Type":"ContainerStarted","Data":"eb14208de003f15a14cb7195fc543177fd9fc0da6e5ccda20ba81950c60b0347"}
Oct 30 04:01:07.262328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:07.261885    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq" event={"ID":"c78b5171-a071-456e-aac4-5834e5b021f3","Type":"ContainerStarted","Data":"f5c8956e259434d26e80ec307f99dd58e292fd3909f5c67c7ee4d0e110c86856"}
Oct 30 04:01:07.301916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:07.301857    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq" podStartSLOduration=2.301839588 podStartE2EDuration="2.301839588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:07.293730336 +0000 UTC m=+3495.722619757" watchObservedRunningTime="2026-10-30 04:01:07.301839588 +0000 UTC m=+3495.730729007"
Oct 30 04:01:08.271116 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:08.271080    4877 generic.go:356] "Generic (PLEG): container finished" podID="c78b5171-a071-456e-aac4-5834e5b021f3" containerID="eb14208de003f15a14cb7195fc543177fd9fc0da6e5ccda20ba81950c60b0347" exitCode=0
Oct 30 04:01:08.271603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:08.271173    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq" event={"ID":"c78b5171-a071-456e-aac4-5834e5b021f3","Type":"ContainerDied","Data":"eb14208de003f15a14cb7195fc543177fd9fc0da6e5ccda20ba81950c60b0347"}
Oct 30 04:01:09.462261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.462237    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:09.568490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.568390    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca\") pod \"c78b5171-a071-456e-aac4-5834e5b021f3\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") "
Oct 30 04:01:09.568658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.568505    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca\") pod \"c78b5171-a071-456e-aac4-5834e5b021f3\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") "
Oct 30 04:01:09.568658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.568622    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm\" (UniqueName: \"kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm\") pod \"c78b5171-a071-456e-aac4-5834e5b021f3\" (UID: \"c78b5171-a071-456e-aac4-5834e5b021f3\") "
Oct 30 04:01:09.568879 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.568844    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca" pod "c78b5171-a071-456e-aac4-5834e5b021f3" (UID: "c78b5171-a071-456e-aac4-5834e5b021f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:09.569091 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.569073    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:09.570674 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.570653    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca" pod "c78b5171-a071-456e-aac4-5834e5b021f3" (UID: "c78b5171-a071-456e-aac4-5834e5b021f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:09.570762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.570654    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm" pod "c78b5171-a071-456e-aac4-5834e5b021f3" (UID: "c78b5171-a071-456e-aac4-5834e5b021f3"). InnerVolumeSpecName "kube-api-access-kl2mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:09.670804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.670767    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2mm\" (UniqueName: \"kubernetes.io/projected/c78b5171-a071-456e-aac4-5834e5b021f3-kube-api-access-kl2mm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:09.670804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:09.670801    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78b5171-a071-456e-aac4-5834e5b021f3-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:10.328555 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:10.327351    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq" event={"ID":"c78b5171-a071-456e-aac4-5834e5b021f3","Type":"ContainerDied","Data":"f5c8956e259434d26e80ec307f99dd58e292fd3909f5c67c7ee4d0e110c86856"}
Oct 30 04:01:10.328555 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:10.327408    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5c8956e259434d26e80ec307f99dd58e292fd3909f5c67c7ee4d0e110c86856"
Oct 30 04:01:10.328555 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:10.327528    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"
Oct 30 04:01:16.230292 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:01:16.230255    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:01:17.302554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.302522    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"]
Oct 30 04:01:17.303371 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.303340    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c78b5171-a071-456e-aac4-5834e5b021f3" containerName="revoke-keys"
Oct 30 04:01:17.303371 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.303368    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78b5171-a071-456e-aac4-5834e5b021f3" containerName="revoke-keys"
Oct 30 04:01:17.303540 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.303530    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78b5171-a071-456e-aac4-5834e5b021f3" containerName="revoke-keys"
Oct 30 04:01:17.306967 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.306945    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.325517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.325496    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"]
Oct 30 04:01:17.329585 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.329562    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:01:17.329721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.329597    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:01:17.346487 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.346405    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtpt\" (UniqueName: \"kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.346685 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.346664    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.346818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.346789    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.448283 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.448244    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.448480 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.448307    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.448480 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.448434    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtpt\" (UniqueName: \"kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.449006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.448981    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.450894 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.450862    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.465265 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.465241    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtpt\" (UniqueName: \"kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt\") pod \"maas-api-revoke-sub-e6f3a1fd-hkjv6\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.621887 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.621808    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:17.925461 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:17.925412    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"]
Oct 30 04:01:17.926920 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:01:17.926891    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dd85c6_fd63_46d7_982b_f8f30b926ba8.slice/crio-d262665127fb2d114c7b4f8b583e11dc147f714a106abcbc7a8be9e7404ce23c WatchSource:0}: Error finding container d262665127fb2d114c7b4f8b583e11dc147f714a106abcbc7a8be9e7404ce23c: Status 404 returned error can't find the container with id d262665127fb2d114c7b4f8b583e11dc147f714a106abcbc7a8be9e7404ce23c
Oct 30 04:01:18.446735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:18.446638    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6" event={"ID":"b8dd85c6-fd63-46d7-982b-f8f30b926ba8","Type":"ContainerStarted","Data":"b6c4b00a2f082a27b1e78a68ea517077fce26e568cce8a0a54c97bdfd133e271"}
Oct 30 04:01:18.446735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:18.446703    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6" event={"ID":"b8dd85c6-fd63-46d7-982b-f8f30b926ba8","Type":"ContainerStarted","Data":"d262665127fb2d114c7b4f8b583e11dc147f714a106abcbc7a8be9e7404ce23c"}
Oct 30 04:01:18.481940 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:18.481881    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6" podStartSLOduration=1.481863569 podStartE2EDuration="1.481863569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:18.479280257 +0000 UTC m=+3506.908169675" watchObservedRunningTime="2026-10-30 04:01:18.481863569 +0000 UTC m=+3506.910752988"
Oct 30 04:01:19.459505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:19.459475    4877 generic.go:356] "Generic (PLEG): container finished" podID="b8dd85c6-fd63-46d7-982b-f8f30b926ba8" containerID="b6c4b00a2f082a27b1e78a68ea517077fce26e568cce8a0a54c97bdfd133e271" exitCode=0
Oct 30 04:01:19.459922 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:19.459566    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6" event={"ID":"b8dd85c6-fd63-46d7-982b-f8f30b926ba8","Type":"ContainerDied","Data":"b6c4b00a2f082a27b1e78a68ea517077fce26e568cce8a0a54c97bdfd133e271"}
Oct 30 04:01:20.649309 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.649278    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:20.699015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.698979    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca\") pod \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") "
Oct 30 04:01:20.699219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.699087    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt\" (UniqueName: \"kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt\") pod \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") "
Oct 30 04:01:20.699521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.699307    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca\") pod \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\" (UID: \"b8dd85c6-fd63-46d7-982b-f8f30b926ba8\") "
Oct 30 04:01:20.700382 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.700353    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca" pod "b8dd85c6-fd63-46d7-982b-f8f30b926ba8" (UID: "b8dd85c6-fd63-46d7-982b-f8f30b926ba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:20.702645 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.702608    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca" pod "b8dd85c6-fd63-46d7-982b-f8f30b926ba8" (UID: "b8dd85c6-fd63-46d7-982b-f8f30b926ba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:20.702989 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.702961    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt" pod "b8dd85c6-fd63-46d7-982b-f8f30b926ba8" (UID: "b8dd85c6-fd63-46d7-982b-f8f30b926ba8"). InnerVolumeSpecName "kube-api-access-qxtpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:20.801472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.801399    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:20.801652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.801476    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtpt\" (UniqueName: \"kubernetes.io/projected/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-kube-api-access-qxtpt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:20.801652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:20.801495    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8dd85c6-fd63-46d7-982b-f8f30b926ba8-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:21.478521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:21.478488    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6" event={"ID":"b8dd85c6-fd63-46d7-982b-f8f30b926ba8","Type":"ContainerDied","Data":"d262665127fb2d114c7b4f8b583e11dc147f714a106abcbc7a8be9e7404ce23c"}
Oct 30 04:01:21.478713 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:21.478540    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d262665127fb2d114c7b4f8b583e11dc147f714a106abcbc7a8be9e7404ce23c"
Oct 30 04:01:21.478713 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:21.478550    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"
Oct 30 04:01:33.357738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:33.357603    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"]
Oct 30 04:01:33.358376 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:33.357849    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy" containerID="cri-o://cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d" gracePeriod=30
Oct 30 04:01:38.729282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.729250    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 04:01:38.799404 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799369    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.799626 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799482    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.799626 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799523    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.799626 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799560    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.799824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799663    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.799824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799745    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.799939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799824    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.800011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.799989    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.800229 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.800209    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy\") pod \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\" (UID: \"636f9bc8-0c75-4a86-8a8e-67d950e052d0\") "
Oct 30 04:01:38.800553 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.800523    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:38.800651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.800601    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:38.800651 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.800615    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:38.801055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.801024    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:38.802018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.801838    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.802018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.801867    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.802018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.801882    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.802018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.801896    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.803056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.803023    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:38.803866 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.803839    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "kube-api-access-ksgm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:38.804823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.804794    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:01:38.805724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.805700    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:38.805841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.805740    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token" pod "636f9bc8-0c75-4a86-8a8e-67d950e052d0" (UID: "636f9bc8-0c75-4a86-8a8e-67d950e052d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:38.886828 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.886793    4877 generic.go:356] "Generic (PLEG): container finished" podID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerID="cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d" exitCode=0
Oct 30 04:01:38.886984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.886858    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"
Oct 30 04:01:38.886984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.886874    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" event={"ID":"636f9bc8-0c75-4a86-8a8e-67d950e052d0","Type":"ContainerDied","Data":"cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d"}
Oct 30 04:01:38.886984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.886932    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d" event={"ID":"636f9bc8-0c75-4a86-8a8e-67d950e052d0","Type":"ContainerDied","Data":"8dcf6813d5b36855763b24d69ffb9fbf3a3eedc1843e426e4af5b79f3da79a97"}
Oct 30 04:01:38.886984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.886956    4877 scope.go:122] "RemoveContainer" containerID="cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d"
Oct 30 04:01:38.900341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.900192    4877 scope.go:122] "RemoveContainer" containerID="8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"
Oct 30 04:01:38.903184 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.903161    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.903278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.903187    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksgm8\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-kube-api-access-ksgm8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.903278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.903203    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.903278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.903215    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636f9bc8-0c75-4a86-8a8e-67d950e052d0-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.903278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.903226    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636f9bc8-0c75-4a86-8a8e-67d950e052d0-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:01:38.914567 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.914543    4877 scope.go:122] "RemoveContainer" containerID="cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d"
Oct 30 04:01:38.914872 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:01:38.914845    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d\": container with ID starting with cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d not found: ID does not exist" containerID="cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d"
Oct 30 04:01:38.914982 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.914881    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d"} err="failed to get container status \"cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d\": rpc error: code = NotFound desc = could not find container \"cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d\": container with ID starting with cf44a435f50b3690ff2f29e748ace5f189857673ed6a104e7a2158d803c0572d not found: ID does not exist"
Oct 30 04:01:38.914982 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.914898    4877 scope.go:122] "RemoveContainer" containerID="8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"
Oct 30 04:01:38.915220 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:01:38.915201    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee\": container with ID starting with 8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee not found: ID does not exist" containerID="8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"
Oct 30 04:01:38.915534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.915228    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee"} err="failed to get container status \"8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee\": rpc error: code = NotFound desc = could not find container \"8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee\": container with ID starting with 8c06b267670bb480280f25e8f90319b7f7dca0d12875d2cb8ad9a30dafcee1ee not found: ID does not exist"
Oct 30 04:01:38.938340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.938195    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"]
Oct 30 04:01:38.952464 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:38.946948    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4206d9-openshift-default-8ccdd7999-nrf5d"]
Oct 30 04:01:39.701080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:39.701043    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" path="/var/lib/kubelet/pods/636f9bc8-0c75-4a86-8a8e-67d950e052d0/volumes"
Oct 30 04:01:46.231352 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:01:46.231307    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:01:56.953558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:56.953526    4877 scope.go:122] "RemoveContainer" containerID="8256089c97342d343e6b8fd0944ca64c87f4c9f4ab44523df8b98226aeb9978e"
Oct 30 04:01:56.966060 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:56.966033    4877 scope.go:122] "RemoveContainer" containerID="290342b791911f931e04112116a391047a7f23e6f0d186568c807792ba129dfe"
Oct 30 04:01:56.977248 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:01:56.977223    4877 scope.go:122] "RemoveContainer" containerID="b1b44e19c37e7188eeef3e144a91e385ea6d4097a048780437b4718edc4371b7"
Oct 30 04:02:08.085749 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.085712    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:08.086192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.085768    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:08.311954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.311752    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"]
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.312881    4877 cpu_manager.go:414] "containerMap: removing container" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy"
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.312909    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy"
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.312932    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b8dd85c6-fd63-46d7-982b-f8f30b926ba8" containerName="revoke-keys"
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.312941    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8dd85c6-fd63-46d7-982b-f8f30b926ba8" containerName="revoke-keys"
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.313244    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy"
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.313266    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy"
Oct 30 04:02:08.315995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.313277    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8dd85c6-fd63-46d7-982b-f8f30b926ba8" containerName="revoke-keys"
Oct 30 04:02:08.317381 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.317343    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.328046 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.328008    4877 generic.go:356] "Generic (PLEG): container finished" podID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerID="9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b" exitCode=0
Oct 30 04:02:08.328186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.328136    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" event={"ID":"bab551f7-2d17-4d22-bc7c-cf694c0089ca","Type":"ContainerDied","Data":"9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"}
Oct 30 04:02:08.328966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.328666    4877 scope.go:122] "RemoveContainer" containerID="9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"
Oct 30 04:02:08.329863 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.329505    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"]
Oct 30 04:02:08.331361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.331325    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:08.331484 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.331413    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:08.331533 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.331521    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:08.333646 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.333627    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:02:08.333805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.333788    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:02:08.445469 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.443353    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.445469 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.443452    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.445469 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.443495    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snlp8\" (UniqueName: \"kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.544853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.544815    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snlp8\" (UniqueName: \"kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.545094 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.545075    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.545194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.545177    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.545931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.545904    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.547844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.547816    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.561576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.561554    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snlp8\" (UniqueName: \"kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8\") pod \"maas-api-revoke-sub-5bcc16f1-8brpz\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.640693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.640607    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:08.954882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:08.954854    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"]
Oct 30 04:02:08.956377 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:02:08.956347    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8837a87_50fc_4fc8_b285_dc34a615dd52.slice/crio-a772f56116393fff1c684b14c7c7f69a90d7a548cfcd01e6d2c28a98062ea463 WatchSource:0}: Error finding container a772f56116393fff1c684b14c7c7f69a90d7a548cfcd01e6d2c28a98062ea463: Status 404 returned error can't find the container with id a772f56116393fff1c684b14c7c7f69a90d7a548cfcd01e6d2c28a98062ea463
Oct 30 04:02:09.339434 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:09.339385    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" event={"ID":"bab551f7-2d17-4d22-bc7c-cf694c0089ca","Type":"ContainerStarted","Data":"98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a"}
Oct 30 04:02:09.340744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:09.340717    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz" event={"ID":"f8837a87-50fc-4fc8-b285-dc34a615dd52","Type":"ContainerStarted","Data":"a772f56116393fff1c684b14c7c7f69a90d7a548cfcd01e6d2c28a98062ea463"}
Oct 30 04:02:09.944856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:09.944818    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:02:09.945276 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:09.945180    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:09.945466 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:09.945439    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:10.450468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:10.449839    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz" event={"ID":"f8837a87-50fc-4fc8-b285-dc34a615dd52","Type":"ContainerStarted","Data":"f982c3a04c542bc737531d43939e6c7ae269b70e4a7c65e96f6fca4ce6877b62"}
Oct 30 04:02:10.941881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:10.941843    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:10.942036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:10.941906    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:11.481448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.476788    4877 generic.go:356] "Generic (PLEG): container finished" podID="f8837a87-50fc-4fc8-b285-dc34a615dd52" containerID="f982c3a04c542bc737531d43939e6c7ae269b70e4a7c65e96f6fca4ce6877b62" exitCode=0
Oct 30 04:02:11.481448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.476836    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz" event={"ID":"f8837a87-50fc-4fc8-b285-dc34a615dd52","Type":"ContainerDied","Data":"f982c3a04c542bc737531d43939e6c7ae269b70e4a7c65e96f6fca4ce6877b62"}
Oct 30 04:02:11.738812 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.738744    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:11.838200 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.838110    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca\") pod \"f8837a87-50fc-4fc8-b285-dc34a615dd52\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") "
Oct 30 04:02:11.847528 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.838385    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8\" (UniqueName: \"kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8\") pod \"f8837a87-50fc-4fc8-b285-dc34a615dd52\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") "
Oct 30 04:02:11.847528 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.838458    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca\") pod \"f8837a87-50fc-4fc8-b285-dc34a615dd52\" (UID: \"f8837a87-50fc-4fc8-b285-dc34a615dd52\") "
Oct 30 04:02:11.847528 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.843290    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca" pod "f8837a87-50fc-4fc8-b285-dc34a615dd52" (UID: "f8837a87-50fc-4fc8-b285-dc34a615dd52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:11.867537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.867495    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca" pod "f8837a87-50fc-4fc8-b285-dc34a615dd52" (UID: "f8837a87-50fc-4fc8-b285-dc34a615dd52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:11.903511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.896723    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8" pod "f8837a87-50fc-4fc8-b285-dc34a615dd52" (UID: "f8837a87-50fc-4fc8-b285-dc34a615dd52"). InnerVolumeSpecName "kube-api-access-snlp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:02:11.941009 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.940977    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:11.941302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.941263    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:11.941821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.941796    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:02:11.941926 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.941823    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snlp8\" (UniqueName: \"kubernetes.io/projected/f8837a87-50fc-4fc8-b285-dc34a615dd52-kube-api-access-snlp8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:02:11.941926 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:11.941841    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8837a87-50fc-4fc8-b285-dc34a615dd52-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:02:12.512417 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:12.512383    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"
Oct 30 04:02:12.512807 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:12.512382    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz" event={"ID":"f8837a87-50fc-4fc8-b285-dc34a615dd52","Type":"ContainerDied","Data":"a772f56116393fff1c684b14c7c7f69a90d7a548cfcd01e6d2c28a98062ea463"}
Oct 30 04:02:12.512807 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:12.512494    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a772f56116393fff1c684b14c7c7f69a90d7a548cfcd01e6d2c28a98062ea463"
Oct 30 04:02:12.941394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:12.941359    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:12.941622 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:12.941413    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:13.941392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:13.941358    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:13.941821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:13.941413    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:14.941276 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:14.941238    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Oct 30 04:02:14.941459 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:14.941302    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Oct 30 04:02:15.914500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:15.914462    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"]
Oct 30 04:02:15.945548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:15.945518    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:02:15.946250 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:15.945984    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:02:15.946782 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:15.946762    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:02:16.230700 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:02:16.230615    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:02:20.915674 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:20.915622    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" podUID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" containerName="payload-processing" containerID="cri-o://830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34" gracePeriod=25
Oct 30 04:02:21.267197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.267172    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 04:02:21.353642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.353608    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn\" (UniqueName: \"kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn\") pod \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") "
Oct 30 04:02:21.353834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.353808    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume\") pod \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\" (UID: \"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0\") "
Oct 30 04:02:21.354286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.354253    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume" pod "5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" (UID: "5fc76012-2888-4c34-bbe8-5a3b97fb2ff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:21.355925 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.355895    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn" pod "5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" (UID: "5fc76012-2888-4c34-bbe8-5a3b97fb2ff0"). InnerVolumeSpecName "kube-api-access-w72mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:02:21.455461 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.455409    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72mn\" (UniqueName: \"kubernetes.io/projected/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-kube-api-access-w72mn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:02:21.455461 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.455455    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:02:21.648384 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.648349    4877 generic.go:356] "Generic (PLEG): container finished" podID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" containerID="830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34" exitCode=0
Oct 30 04:02:21.648610 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.648407    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"
Oct 30 04:02:21.648610 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.648465    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" event={"ID":"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0","Type":"ContainerDied","Data":"830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34"}
Oct 30 04:02:21.648610 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.648520    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" event={"ID":"5fc76012-2888-4c34-bbe8-5a3b97fb2ff0","Type":"ContainerDied","Data":"5f8f4287abac794646c67adf6de4e20e2c42ce7e07a9a6979d20799b3ea707db"}
Oct 30 04:02:21.648610 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.648543    4877 scope.go:122] "RemoveContainer" containerID="830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34"
Oct 30 04:02:21.660416 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.660400    4877 scope.go:122] "RemoveContainer" containerID="830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34"
Oct 30 04:02:21.660758 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:02:21.660732    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34\": container with ID starting with 830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34 not found: ID does not exist" containerID="830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34"
Oct 30 04:02:21.660839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.660768    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34"} err="failed to get container status \"830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34\": rpc error: code = NotFound desc = could not find container \"830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34\": container with ID starting with 830bd97bacd3e513aa69ab52053383d440905517f5fc70a59bc06162440c1c34 not found: ID does not exist"
Oct 30 04:02:21.701987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.701957    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"]
Oct 30 04:02:21.702887 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:21.702864    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6"]
Oct 30 04:02:22.215669 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:22.215618    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b267f8-799f4d5476-r2dh6" podUID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.27:50052\" within 1s: context deadline exceeded"
Oct 30 04:02:22.216093 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:02:22.215753    4877 logging.go:55] [core] [Channel #14169 SubChannel #14170]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.27:50052", ServerName: "10.133.1.27:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.27:50052: operation was canceled"
Oct 30 04:02:23.700624 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:23.700578    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" path="/var/lib/kubelet/pods/5fc76012-2888-4c34-bbe8-5a3b97fb2ff0/volumes"
Oct 30 04:02:46.361463 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:02:46.360569    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:02:52.958997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:52.958968    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:02:52.973970 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:52.973946    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:02:56.854679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.854645    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"]
Oct 30 04:02:56.855400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855383    4877 cpu_manager.go:414] "containerMap: removing container" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy"
Oct 30 04:02:56.855510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855403    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636f9bc8-0c75-4a86-8a8e-67d950e052d0" containerName="istio-proxy"
Oct 30 04:02:56.855510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855432    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" containerName="payload-processing"
Oct 30 04:02:56.855510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855438    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" containerName="payload-processing"
Oct 30 04:02:56.855510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855509    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f8837a87-50fc-4fc8-b285-dc34a615dd52" containerName="revoke-keys"
Oct 30 04:02:56.855725 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855517    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8837a87-50fc-4fc8-b285-dc34a615dd52" containerName="revoke-keys"
Oct 30 04:02:56.855725 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855643    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8837a87-50fc-4fc8-b285-dc34a615dd52" containerName="revoke-keys"
Oct 30 04:02:56.855725 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.855654    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc76012-2888-4c34-bbe8-5a3b97fb2ff0" containerName="payload-processing"
Oct 30 04:02:56.859699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.859677    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:56.885086 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.885053    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"]
Oct 30 04:02:56.920650 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:56.920614    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e1a0c2-openshift-default-dockercfg-njtkh\""
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.023281    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.023392    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.023449    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qdn\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.023496    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.023531    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.025652    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.025753    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.025877    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.033516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.025949    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127346 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127260    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127375    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127410    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qdn\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127493    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127624    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127644    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127765    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.127954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.127865    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.128051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.128025    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.128121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.128060    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.128121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.128105    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.128235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.128175    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.128552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.128526    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.128688 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.128651    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.130132 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.130111    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.130964 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.130942    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.144554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.144527    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.144957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.144933    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qdn\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.176745 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.176722    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:02:57.335807 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.335753    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"]
Oct 30 04:02:57.336043 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.335992    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy" containerID="cri-o://ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d" gracePeriod=30
Oct 30 04:02:57.596010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.589355    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"]
Oct 30 04:02:57.623193 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:02:57.622537    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26db8af_a6b2_412d_87a6_e733209c41f2.slice/crio-432d7bf730afd5bd9121e31ac86791e6dd586af338235efe538b75353e8576c0 WatchSource:0}: Error finding container 432d7bf730afd5bd9121e31ac86791e6dd586af338235efe538b75353e8576c0: Status 404 returned error can't find the container with id 432d7bf730afd5bd9121e31ac86791e6dd586af338235efe538b75353e8576c0
Oct 30 04:02:57.628339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.627627    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:57.628339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.627699    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:57.628339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:57.627728    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:02:58.328274 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:58.328235    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" event={"ID":"e26db8af-a6b2-412d-87a6-e733209c41f2","Type":"ContainerStarted","Data":"70ba340a460928d45f420220846f8f48fc141b40d1fd736b4a61672479ac1c3d"}
Oct 30 04:02:58.328744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:58.328292    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" event={"ID":"e26db8af-a6b2-412d-87a6-e733209c41f2","Type":"ContainerStarted","Data":"432d7bf730afd5bd9121e31ac86791e6dd586af338235efe538b75353e8576c0"}
Oct 30 04:02:58.372686 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:58.372633    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podStartSLOduration=2.372614726 podStartE2EDuration="2.372614726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:58.367457929 +0000 UTC m=+3606.796347347" watchObservedRunningTime="2026-10-30 04:02:58.372614726 +0000 UTC m=+3606.801504140"
Oct 30 04:02:59.198445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:59.183597    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 30 04:02:59.198445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:59.183669    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 30 04:02:59.198445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:02:59.183772    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:03:00.180858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:00.180816    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 30 04:03:00.181518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:00.181480    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 30 04:03:01.187871 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:01.187831    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 30 04:03:01.188499 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:01.188462    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 30 04:03:02.184449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.180855    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 30 04:03:02.184449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.180942    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 30 04:03:02.719789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.719703    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 04:03:02.766798 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.766767    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.766995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.766827    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.766995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.766954    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.766995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.766987    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.767176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767026    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.767176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767084    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.767176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767108    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:02.767352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767194    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.767352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767264    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.767352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767310    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo\") pod \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\" (UID: \"c7b8fedb-3066-4c44-8e86-d3f72dee551a\") "
Oct 30 04:03:02.767571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767530    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:02.767751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767605    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:02.767932 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.767908    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:02.768132 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.768100    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.768132 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.768120    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.768285 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.768135    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.768285 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.768149    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.768285 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.768222    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:02.770218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.770182    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:03:02.776463 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.771827    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:02.776463 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.771920    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:02.776463 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.771942    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z" pod "c7b8fedb-3066-4c44-8e86-d3f72dee551a" (UID: "c7b8fedb-3066-4c44-8e86-d3f72dee551a"). InnerVolumeSpecName "kube-api-access-tjx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:02.925518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.872736    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.925518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.872778    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjx4z\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-kube-api-access-tjx4z\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.925518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.872795    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.925518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.872809    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:02.925518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:02.872822    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b8fedb-3066-4c44-8e86-d3f72dee551a-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:03:03.185640 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.185580    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 30 04:03:03.186047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.185928    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 30 04:03:03.464387 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.464298    4877 generic.go:356] "Generic (PLEG): container finished" podID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerID="ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d" exitCode=0
Oct 30 04:03:03.464387 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.464349    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"
Oct 30 04:03:03.464619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.464386    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" event={"ID":"c7b8fedb-3066-4c44-8e86-d3f72dee551a","Type":"ContainerDied","Data":"ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d"}
Oct 30 04:03:03.464619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.464459    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg" event={"ID":"c7b8fedb-3066-4c44-8e86-d3f72dee551a","Type":"ContainerDied","Data":"0172f2195743f99ba14b96f37457a620137ec830aa2aa282f6019fd3aa4aaf46"}
Oct 30 04:03:03.464619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.464489    4877 scope.go:122] "RemoveContainer" containerID="ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d"
Oct 30 04:03:03.500644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.500272    4877 scope.go:122] "RemoveContainer" containerID="f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"
Oct 30 04:03:03.523997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.523964    4877 scope.go:122] "RemoveContainer" containerID="ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d"
Oct 30 04:03:03.524390 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:03:03.524356    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d\": container with ID starting with ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d not found: ID does not exist" containerID="ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d"
Oct 30 04:03:03.524505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.524399    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d"} err="failed to get container status \"ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d\": rpc error: code = NotFound desc = could not find container \"ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d\": container with ID starting with ebd0184c29f467617f38cbabd7fb26afd904347ed1b4db2dc5475087fbbe956d not found: ID does not exist"
Oct 30 04:03:03.524505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.524419    4877 scope.go:122] "RemoveContainer" containerID="f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"
Oct 30 04:03:03.524731 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:03:03.524713    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9\": container with ID starting with f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9 not found: ID does not exist" containerID="f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"
Oct 30 04:03:03.524799 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.524735    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9"} err="failed to get container status \"f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9\": rpc error: code = NotFound desc = could not find container \"f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9\": container with ID starting with f979cee05bd0ebb5aa1f2467097a0d33a27a93f699d68eb7262c6988bdfd55f9 not found: ID does not exist"
Oct 30 04:03:03.551833 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.551802    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"]
Oct 30 04:03:03.564682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.564654    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-c4lfg"]
Oct 30 04:03:03.716997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:03.716892    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" path="/var/lib/kubelet/pods/c7b8fedb-3066-4c44-8e86-d3f72dee551a/volumes"
Oct 30 04:03:04.177341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:04.177299    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Oct 30 04:03:04.177802 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:04.177366    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Oct 30 04:03:05.183465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:05.183400    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:03:05.184053 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:05.184026    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:03:05.184925 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:05.184894    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:03:12.026740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:12.026703    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"]
Oct 30 04:03:12.035860 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:12.035829    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ca44c-bg29k"]
Oct 30 04:03:13.702204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:13.702169    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dadb0c55-dc64-42c6-b82f-99005c95ab47" path="/var/lib/kubelet/pods/dadb0c55-dc64-42c6-b82f-99005c95ab47/volumes"
Oct 30 04:03:16.232458 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:03:16.232409    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:03:31.028131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:31.028092    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"]
Oct 30 04:03:31.035504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:31.035473    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75a701a1-z8sgp"]
Oct 30 04:03:31.705282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:31.705243    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30dd828-bfb1-4404-b2c3-0cd07375c56c" path="/var/lib/kubelet/pods/d30dd828-bfb1-4404-b2c3-0cd07375c56c/volumes"
Oct 30 04:03:43.029911 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:43.029830    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"]
Oct 30 04:03:43.039320 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:43.039289    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74884c5b-7c2xx"]
Oct 30 04:03:43.701389 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:03:43.701345    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace82714-9e24-49c7-a293-6fa9c027f79d" path="/var/lib/kubelet/pods/ace82714-9e24-49c7-a293-6fa9c027f79d/volumes"
Oct 30 04:03:46.232160 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:03:46.232123    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:04:14.235858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:14.235817    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"]
Oct 30 04:04:14.236567 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:14.236100    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" containerID="cri-o://98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a" gracePeriod=30
Oct 30 04:04:15.948399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:15.948361    4877 patch_prober.go:29] interesting pod/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:04:15.948815 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:15.948438    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:04:16.230165 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:04:16.230068    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:04:19.544473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.544447    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:04:19.607179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607146    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.607179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607187    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.607456 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607263    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607469    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607608    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607763    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607824    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607858    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.607916    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608019    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608074    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket\") pod \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\" (UID: \"bab551f7-2d17-4d22-bc7c-cf694c0089ca\") "
Oct 30 04:04:19.608724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608261    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:19.608724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608568    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:04:19.608843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608774    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:04:19.608917 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608892    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.608986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608918    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.608986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608935    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.608986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608951    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.608986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.608966    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.610306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.610238    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:19.610670 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.610648    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:04:19.611171 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.611140    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "kube-api-access-94v5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:19.611274 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.611206    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo" pod "bab551f7-2d17-4d22-bc7c-cf694c0089ca" (UID: "bab551f7-2d17-4d22-bc7c-cf694c0089ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:04:19.709685 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.709659    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.709685 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.709685    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.709820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.709695    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94v5d\" (UniqueName: \"kubernetes.io/projected/bab551f7-2d17-4d22-bc7c-cf694c0089ca-kube-api-access-94v5d\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:19.709820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:19.709704    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bab551f7-2d17-4d22-bc7c-cf694c0089ca-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:20.287726 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.287686    4877 generic.go:356] "Generic (PLEG): container finished" podID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerID="98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a" exitCode=0
Oct 30 04:04:20.287916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.287741    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"
Oct 30 04:04:20.287916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.287772    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" event={"ID":"bab551f7-2d17-4d22-bc7c-cf694c0089ca","Type":"ContainerDied","Data":"98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a"}
Oct 30 04:04:20.287916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.287853    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh" event={"ID":"bab551f7-2d17-4d22-bc7c-cf694c0089ca","Type":"ContainerDied","Data":"4953798ee2aa7192d3aa3ed1d468ec8cda31ac6a7cd54273bc27791ebb957fb2"}
Oct 30 04:04:20.287916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.287886    4877 scope.go:122] "RemoveContainer" containerID="98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a"
Oct 30 04:04:20.299465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.299431    4877 scope.go:122] "RemoveContainer" containerID="9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"
Oct 30 04:04:20.310223 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.310202    4877 scope.go:122] "RemoveContainer" containerID="98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a"
Oct 30 04:04:20.310580 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:04:20.310549    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a\": container with ID starting with 98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a not found: ID does not exist" containerID="98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a"
Oct 30 04:04:20.310693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.310589    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a"} err="failed to get container status \"98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a\": rpc error: code = NotFound desc = could not find container \"98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a\": container with ID starting with 98430b7706751c3c64b41c975bb0b21d39d533a8278f09fc6bf49d2beaec969a not found: ID does not exist"
Oct 30 04:04:20.310693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.310615    4877 scope.go:122] "RemoveContainer" containerID="9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"
Oct 30 04:04:20.310925 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:04:20.310904    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b\": container with ID starting with 9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b not found: ID does not exist" containerID="9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"
Oct 30 04:04:20.310995 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.310936    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b"} err="failed to get container status \"9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b\": rpc error: code = NotFound desc = could not find container \"9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b\": container with ID starting with 9116d4d2cc7001e56d47b4f03bd66c1897a630da586f8f2c4b31231a9d0d8b1b not found: ID does not exist"
Oct 30 04:04:20.347121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.347087    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"]
Oct 30 04:04:20.353850 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:20.353827    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1597f8bc-openshift-default-97d6dc64f-7hdkh"]
Oct 30 04:04:21.700497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:21.700463    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" path="/var/lib/kubelet/pods/bab551f7-2d17-4d22-bc7c-cf694c0089ca/volumes"
Oct 30 04:04:38.021864 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.021817    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"]
Oct 30 04:04:38.023687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.023654    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy"
Oct 30 04:04:38.023687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.023686    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy"
Oct 30 04:04:38.023898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.023711    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy"
Oct 30 04:04:38.023898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.023719    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy"
Oct 30 04:04:38.023898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.023812    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy"
Oct 30 04:04:38.023898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.023820    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy"
Oct 30 04:04:38.024247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.024223    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy"
Oct 30 04:04:38.024348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.024281    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy"
Oct 30 04:04:38.024348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.024302    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab551f7-2d17-4d22-bc7c-cf694c0089ca" containerName="istio-proxy"
Oct 30 04:04:38.030508 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.029871    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.033757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.033725    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"]
Oct 30 04:04:38.093002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.092958    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:04:38.096326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.096295    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:04:38.179510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.179474    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.179715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.179635    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.179715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.179688    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdn58\" (UniqueName: \"kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.281576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.281484    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.281576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.281557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.281815 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.281614    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdn58\" (UniqueName: \"kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.282372 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.282350    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.284200 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.284174    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.301834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.301807    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdn58\" (UniqueName: \"kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58\") pod \"maas-api-revoke-sub-8383a775-zxtdk\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.358226 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.358195    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:38.682098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:38.682071    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"]
Oct 30 04:04:38.684999 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:04:38.684960    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf409fe59_079b_49a9_992c_9bd76d3db66d.slice/crio-30117eb6077902c78b7cf3dd30f930c88cde122b689539bb54b6ca6f3c5494f1 WatchSource:0}: Error finding container 30117eb6077902c78b7cf3dd30f930c88cde122b689539bb54b6ca6f3c5494f1: Status 404 returned error can't find the container with id 30117eb6077902c78b7cf3dd30f930c88cde122b689539bb54b6ca6f3c5494f1
Oct 30 04:04:39.509315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:39.509254    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk" event={"ID":"f409fe59-079b-49a9-992c-9bd76d3db66d","Type":"ContainerStarted","Data":"d4593f59fe4453123833c4570a6879cb13d167c9fc1033ad4c255aa4c27de8fb"}
Oct 30 04:04:39.509850 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:39.509326    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk" event={"ID":"f409fe59-079b-49a9-992c-9bd76d3db66d","Type":"ContainerStarted","Data":"30117eb6077902c78b7cf3dd30f930c88cde122b689539bb54b6ca6f3c5494f1"}
Oct 30 04:04:39.547287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:39.547237    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk" podStartSLOduration=2.547224215 podStartE2EDuration="2.547224215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:39.542385375 +0000 UTC m=+3707.971274793" watchObservedRunningTime="2026-10-30 04:04:39.547224215 +0000 UTC m=+3707.976113634"
Oct 30 04:04:40.526350 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:40.526315    4877 generic.go:356] "Generic (PLEG): container finished" podID="f409fe59-079b-49a9-992c-9bd76d3db66d" containerID="d4593f59fe4453123833c4570a6879cb13d167c9fc1033ad4c255aa4c27de8fb" exitCode=0
Oct 30 04:04:40.526826 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:40.526370    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk" event={"ID":"f409fe59-079b-49a9-992c-9bd76d3db66d","Type":"ContainerDied","Data":"d4593f59fe4453123833c4570a6879cb13d167c9fc1033ad4c255aa4c27de8fb"}
Oct 30 04:04:41.735045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.735019    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:41.836279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.836181    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58\" (UniqueName: \"kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58\") pod \"f409fe59-079b-49a9-992c-9bd76d3db66d\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") "
Oct 30 04:04:41.836279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.836241    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca\") pod \"f409fe59-079b-49a9-992c-9bd76d3db66d\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") "
Oct 30 04:04:41.836570 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.836378    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca\") pod \"f409fe59-079b-49a9-992c-9bd76d3db66d\" (UID: \"f409fe59-079b-49a9-992c-9bd76d3db66d\") "
Oct 30 04:04:41.836966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.836936    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca" pod "f409fe59-079b-49a9-992c-9bd76d3db66d" (UID: "f409fe59-079b-49a9-992c-9bd76d3db66d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:41.837417 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.837394    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:41.839504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.839475    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58" pod "f409fe59-079b-49a9-992c-9bd76d3db66d" (UID: "f409fe59-079b-49a9-992c-9bd76d3db66d"). InnerVolumeSpecName "kube-api-access-gdn58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:41.842759 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.842733    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca" pod "f409fe59-079b-49a9-992c-9bd76d3db66d" (UID: "f409fe59-079b-49a9-992c-9bd76d3db66d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:41.938289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.938252    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdn58\" (UniqueName: \"kubernetes.io/projected/f409fe59-079b-49a9-992c-9bd76d3db66d-kube-api-access-gdn58\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:41.938289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:41.938283    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f409fe59-079b-49a9-992c-9bd76d3db66d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:42.545102 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.545068    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"
Oct 30 04:04:42.545286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.545096    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk" event={"ID":"f409fe59-079b-49a9-992c-9bd76d3db66d","Type":"ContainerDied","Data":"30117eb6077902c78b7cf3dd30f930c88cde122b689539bb54b6ca6f3c5494f1"}
Oct 30 04:04:42.545286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.545177    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30117eb6077902c78b7cf3dd30f930c88cde122b689539bb54b6ca6f3c5494f1"
Oct 30 04:04:42.892505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.892410    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"]
Oct 30 04:04:42.893573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.893551    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f409fe59-079b-49a9-992c-9bd76d3db66d" containerName="revoke-keys"
Oct 30 04:04:42.893650 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.893576    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f409fe59-079b-49a9-992c-9bd76d3db66d" containerName="revoke-keys"
Oct 30 04:04:42.893650 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.893613    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy"
Oct 30 04:04:42.893650 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.893620    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy"
Oct 30 04:04:42.893827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.893814    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b8fedb-3066-4c44-8e86-d3f72dee551a" containerName="istio-proxy"
Oct 30 04:04:42.893862 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.893837    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f409fe59-079b-49a9-992c-9bd76d3db66d" containerName="revoke-keys"
Oct 30 04:04:42.897815 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.897794    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:42.912220 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.912197    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:04:42.913732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.913703    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"]
Oct 30 04:04:42.916860 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.916839    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:04:42.952792 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.952761    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:42.952950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.952837    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:42.953008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:42.952992    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq72k\" (UniqueName: \"kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.029315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.029279    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"]
Oct 30 04:04:43.033599 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.033574    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1194c82-2djkl"]
Oct 30 04:04:43.053969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.053926    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.054146 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.054093    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.054271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.054257    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq72k\" (UniqueName: \"kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.054855 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.054826    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.056508 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.056485    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.071206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.071181    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq72k\" (UniqueName: \"kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k\") pod \"maas-api-revoke-sub-971206c4-lhgb4\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.214321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.214229    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:43.519845 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.519810    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"]
Oct 30 04:04:43.520929 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:04:43.520897    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa145b50_9bbb_4b69_94fa_c9b8c7e5fdf3.slice/crio-c2973f8bba72969f14c0216886070b206c222b88ca8593774abd56629a360204 WatchSource:0}: Error finding container c2973f8bba72969f14c0216886070b206c222b88ca8593774abd56629a360204: Status 404 returned error can't find the container with id c2973f8bba72969f14c0216886070b206c222b88ca8593774abd56629a360204
Oct 30 04:04:43.554234 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.554203    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4" event={"ID":"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3","Type":"ContainerStarted","Data":"c2973f8bba72969f14c0216886070b206c222b88ca8593774abd56629a360204"}
Oct 30 04:04:43.701892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:43.701850    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dc673d-8c55-466a-af7f-b6f6033c7bab" path="/var/lib/kubelet/pods/57dc673d-8c55-466a-af7f-b6f6033c7bab/volumes"
Oct 30 04:04:44.564174 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:44.564132    4877 generic.go:356] "Generic (PLEG): container finished" podID="fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" containerID="be7732ab9f2b7e96be3527fb89eb4c2fec556899257aeb47a4b519e68deaf4cf" exitCode=0
Oct 30 04:04:44.564594 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:44.564217    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4" event={"ID":"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3","Type":"ContainerDied","Data":"be7732ab9f2b7e96be3527fb89eb4c2fec556899257aeb47a4b519e68deaf4cf"}
Oct 30 04:04:45.782545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.782513    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:45.911659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.911556    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca\") pod \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") "
Oct 30 04:04:45.911659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.911613    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k\" (UniqueName: \"kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k\") pod \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") "
Oct 30 04:04:45.911906 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.911844    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca\") pod \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\" (UID: \"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3\") "
Oct 30 04:04:45.912051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.912022    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca" pod "fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" (UID: "fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:45.912452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.912416    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:45.913943 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.913920    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca" pod "fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" (UID: "fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:45.914038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:45.913941    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k" pod "fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" (UID: "fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3"). InnerVolumeSpecName "kube-api-access-rq72k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:46.014331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:46.014296    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq72k\" (UniqueName: \"kubernetes.io/projected/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-kube-api-access-rq72k\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:46.014331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:46.014329    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:46.230265 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:04:46.230185    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:04:46.584038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:46.584004    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"
Oct 30 04:04:46.584038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:46.584033    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4" event={"ID":"fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3","Type":"ContainerDied","Data":"c2973f8bba72969f14c0216886070b206c222b88ca8593774abd56629a360204"}
Oct 30 04:04:46.584323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:46.584086    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2973f8bba72969f14c0216886070b206c222b88ca8593774abd56629a360204"
Oct 30 04:04:48.359596 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.359557    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"]
Oct 30 04:04:48.360369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.360350    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" containerName="revoke-keys"
Oct 30 04:04:48.360449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.360371    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" containerName="revoke-keys"
Oct 30 04:04:48.360509 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.360499    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" containerName="revoke-keys"
Oct 30 04:04:48.364514 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.364482    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.385518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.385480    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"]
Oct 30 04:04:48.386994 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.386969    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:04:48.387122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.386994    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:04:48.484446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.480602    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.484446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.480778    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljtn\" (UniqueName: \"kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.484446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.480844    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.582361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.582325    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.582624 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.582483    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljtn\" (UniqueName: \"kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.582624 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.582520    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.583280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.583259    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.584872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.584847    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.600120 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.600087    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljtn\" (UniqueName: \"kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn\") pod \"maas-api-revoke-sub-fa8919eb-q2r99\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.680340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.680257    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:48.989016 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:04:48.988976    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0134590_8140_487c_8286_3f663d7d942b.slice/crio-112d43ae99643e27d86be872d3d1100632032f799a07ae0b1e334dcf0fc4f016 WatchSource:0}: Error finding container 112d43ae99643e27d86be872d3d1100632032f799a07ae0b1e334dcf0fc4f016: Status 404 returned error can't find the container with id 112d43ae99643e27d86be872d3d1100632032f799a07ae0b1e334dcf0fc4f016
Oct 30 04:04:48.989371 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:48.989340    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"]
Oct 30 04:04:49.613832 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:49.613792    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99" event={"ID":"f0134590-8140-487c-8286-3f663d7d942b","Type":"ContainerStarted","Data":"e1d990b5bb04d9cb96e3a485163360531dfc942431a9236deeb875910a0cda88"}
Oct 30 04:04:49.614207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:49.613914    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99" event={"ID":"f0134590-8140-487c-8286-3f663d7d942b","Type":"ContainerStarted","Data":"112d43ae99643e27d86be872d3d1100632032f799a07ae0b1e334dcf0fc4f016"}
Oct 30 04:04:49.648590 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:49.648477    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99" podStartSLOduration=1.6484602000000002 podStartE2EDuration="1.6484602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:49.646500421 +0000 UTC m=+3718.075389840" watchObservedRunningTime="2026-10-30 04:04:49.6484602 +0000 UTC m=+3718.077349618"
Oct 30 04:04:50.624123 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:50.624080    4877 generic.go:356] "Generic (PLEG): container finished" podID="f0134590-8140-487c-8286-3f663d7d942b" containerID="e1d990b5bb04d9cb96e3a485163360531dfc942431a9236deeb875910a0cda88" exitCode=0
Oct 30 04:04:50.624774 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:50.624128    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99" event={"ID":"f0134590-8140-487c-8286-3f663d7d942b","Type":"ContainerDied","Data":"e1d990b5bb04d9cb96e3a485163360531dfc942431a9236deeb875910a0cda88"}
Oct 30 04:04:51.823196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.823170    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:51.934614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.934524    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca\") pod \"f0134590-8140-487c-8286-3f663d7d942b\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") "
Oct 30 04:04:51.934614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.934572    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn\" (UniqueName: \"kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn\") pod \"f0134590-8140-487c-8286-3f663d7d942b\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") "
Oct 30 04:04:51.934826 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.934801    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca\") pod \"f0134590-8140-487c-8286-3f663d7d942b\" (UID: \"f0134590-8140-487c-8286-3f663d7d942b\") "
Oct 30 04:04:51.934910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.934876    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca" pod "f0134590-8140-487c-8286-3f663d7d942b" (UID: "f0134590-8140-487c-8286-3f663d7d942b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:51.935501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.935476    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0134590-8140-487c-8286-3f663d7d942b-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:51.936839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.936813    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca" pod "f0134590-8140-487c-8286-3f663d7d942b" (UID: "f0134590-8140-487c-8286-3f663d7d942b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:51.936944 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:51.936840    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn" pod "f0134590-8140-487c-8286-3f663d7d942b" (UID: "f0134590-8140-487c-8286-3f663d7d942b"). InnerVolumeSpecName "kube-api-access-6ljtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:52.036298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:52.036269    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ljtn\" (UniqueName: \"kubernetes.io/projected/f0134590-8140-487c-8286-3f663d7d942b-kube-api-access-6ljtn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:52.036298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:52.036297    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0134590-8140-487c-8286-3f663d7d942b-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:52.650214 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:52.650179    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"
Oct 30 04:04:52.650412 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:52.650213    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99" event={"ID":"f0134590-8140-487c-8286-3f663d7d942b","Type":"ContainerDied","Data":"112d43ae99643e27d86be872d3d1100632032f799a07ae0b1e334dcf0fc4f016"}
Oct 30 04:04:52.650412 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:52.650273    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112d43ae99643e27d86be872d3d1100632032f799a07ae0b1e334dcf0fc4f016"
Oct 30 04:04:53.232329 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.232292    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"]
Oct 30 04:04:53.233122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.233102    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f0134590-8140-487c-8286-3f663d7d942b" containerName="revoke-keys"
Oct 30 04:04:53.233172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.233123    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0134590-8140-487c-8286-3f663d7d942b" containerName="revoke-keys"
Oct 30 04:04:53.233243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.233234    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0134590-8140-487c-8286-3f663d7d942b" containerName="revoke-keys"
Oct 30 04:04:53.237034 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.237017    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.256128 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.256091    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"]
Oct 30 04:04:53.259699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.259665    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:04:53.259835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.259731    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:04:53.356540 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.356500    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nn2\" (UniqueName: \"kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.356738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.356656    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.356738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.356725    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.458265 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.458231    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nn2\" (UniqueName: \"kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.458448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.458332    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.458448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.458399    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.459129 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.459108    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.460827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.460805    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.479486 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.479453    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nn2\" (UniqueName: \"kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2\") pod \"maas-api-revoke-sub-958b9133-8j2kd\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.554526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.554414    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:53.868900 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:53.868865    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"]
Oct 30 04:04:53.869413 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:04:53.869387    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9206ed1_6957_4cea_954c_fe94b0a43d42.slice/crio-732e19729ae779fdfa1a36aa0641599cd82e76045d4cb01eca19549c1447db63 WatchSource:0}: Error finding container 732e19729ae779fdfa1a36aa0641599cd82e76045d4cb01eca19549c1447db63: Status 404 returned error can't find the container with id 732e19729ae779fdfa1a36aa0641599cd82e76045d4cb01eca19549c1447db63
Oct 30 04:04:54.030582 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:54.030540    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"]
Oct 30 04:04:54.035083 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:54.035052    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e5af3b-nvnnm"]
Oct 30 04:04:54.680810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:54.677663    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd" event={"ID":"d9206ed1-6957-4cea-954c-fe94b0a43d42","Type":"ContainerStarted","Data":"451274e18b8d0549d5a68fc8da14e6c84c9faf43ada7fef9ba969bfe2d32a567"}
Oct 30 04:04:54.680810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:54.677788    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd" event={"ID":"d9206ed1-6957-4cea-954c-fe94b0a43d42","Type":"ContainerStarted","Data":"732e19729ae779fdfa1a36aa0641599cd82e76045d4cb01eca19549c1447db63"}
Oct 30 04:04:54.714941 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:54.714871    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd" podStartSLOduration=1.7148546850000002 podStartE2EDuration="1.714854685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:54.71126426 +0000 UTC m=+3723.140153677" watchObservedRunningTime="2026-10-30 04:04:54.714854685 +0000 UTC m=+3723.143744105"
Oct 30 04:04:55.688260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:55.688225    4877 generic.go:356] "Generic (PLEG): container finished" podID="d9206ed1-6957-4cea-954c-fe94b0a43d42" containerID="451274e18b8d0549d5a68fc8da14e6c84c9faf43ada7fef9ba969bfe2d32a567" exitCode=0
Oct 30 04:04:55.688736 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:55.688312    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd" event={"ID":"d9206ed1-6957-4cea-954c-fe94b0a43d42","Type":"ContainerDied","Data":"451274e18b8d0549d5a68fc8da14e6c84c9faf43ada7fef9ba969bfe2d32a567"}
Oct 30 04:04:55.701500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:55.701470    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837d41e7-4a80-4017-9fb1-e5ca7b612a0d" path="/var/lib/kubelet/pods/837d41e7-4a80-4017-9fb1-e5ca7b612a0d/volumes"
Oct 30 04:04:56.890941 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.890914    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:56.916692 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.916662    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca\") pod \"d9206ed1-6957-4cea-954c-fe94b0a43d42\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") "
Oct 30 04:04:56.916830 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.916728    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2\" (UniqueName: \"kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2\") pod \"d9206ed1-6957-4cea-954c-fe94b0a43d42\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") "
Oct 30 04:04:56.916884 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.916858    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca\") pod \"d9206ed1-6957-4cea-954c-fe94b0a43d42\" (UID: \"d9206ed1-6957-4cea-954c-fe94b0a43d42\") "
Oct 30 04:04:56.917122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.917094    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca" pod "d9206ed1-6957-4cea-954c-fe94b0a43d42" (UID: "d9206ed1-6957-4cea-954c-fe94b0a43d42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:56.917525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.917501    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:56.918946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.918922    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca" pod "d9206ed1-6957-4cea-954c-fe94b0a43d42" (UID: "d9206ed1-6957-4cea-954c-fe94b0a43d42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:56.919041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:56.918961    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2" pod "d9206ed1-6957-4cea-954c-fe94b0a43d42" (UID: "d9206ed1-6957-4cea-954c-fe94b0a43d42"). InnerVolumeSpecName "kube-api-access-z4nn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:57.019513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.019409    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nn2\" (UniqueName: \"kubernetes.io/projected/d9206ed1-6957-4cea-954c-fe94b0a43d42-kube-api-access-z4nn2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:57.019513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.019466    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9206ed1-6957-4cea-954c-fe94b0a43d42-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:04:57.090173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.090142    4877 scope.go:122] "RemoveContainer" containerID="e8029b919c9c251ae245b77f682992206e4f20b32c43c9931db537f21d4d3b7d"
Oct 30 04:04:57.104107 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.104089    4877 scope.go:122] "RemoveContainer" containerID="47662e0f70d09a5b0bfaaf11e42fa4fe95c49646f1e7e30441c2eb69090c5fd1"
Oct 30 04:04:57.114342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.114318    4877 scope.go:122] "RemoveContainer" containerID="c3aa580b6fde3ae89f00288f0381a3a34f22e2af6b30b95500936cb3af828c53"
Oct 30 04:04:57.715488 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.715445    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd" event={"ID":"d9206ed1-6957-4cea-954c-fe94b0a43d42","Type":"ContainerDied","Data":"732e19729ae779fdfa1a36aa0641599cd82e76045d4cb01eca19549c1447db63"}
Oct 30 04:04:57.715715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.715512    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732e19729ae779fdfa1a36aa0641599cd82e76045d4cb01eca19549c1447db63"
Oct 30 04:04:57.715715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:57.715533    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"
Oct 30 04:04:59.237071 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.237032    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"]
Oct 30 04:04:59.238051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.238027    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d9206ed1-6957-4cea-954c-fe94b0a43d42" containerName="revoke-keys"
Oct 30 04:04:59.238051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.238051    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9206ed1-6957-4cea-954c-fe94b0a43d42" containerName="revoke-keys"
Oct 30 04:04:59.238264 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.238250    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9206ed1-6957-4cea-954c-fe94b0a43d42" containerName="revoke-keys"
Oct 30 04:04:59.242164 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.242145    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.259929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.259903    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"]
Oct 30 04:04:59.263685 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.263665    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:04:59.263910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.263885    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:04:59.352502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.352466    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.352684 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.352603    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpjk\" (UniqueName: \"kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.352729 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.352687    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.454049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.454013    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.454272 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.454145    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpjk\" (UniqueName: \"kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.454272 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.454240    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.454915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.454883    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.456647 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.456621    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.471902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.471870    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpjk\" (UniqueName: \"kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk\") pod \"maas-api-revoke-sub-6af3c4c3-7tqsm\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.556689 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.556660    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:04:59.869985 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:04:59.869926    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"]
Oct 30 04:04:59.870122 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:04:59.869978    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4092798d_6c7a_4dd4_98e1_306a079f0ff6.slice/crio-c3607f83fff4f6b4a88eefab424976a3534177f1c8e64caed166692257a7151a WatchSource:0}: Error finding container c3607f83fff4f6b4a88eefab424976a3534177f1c8e64caed166692257a7151a: Status 404 returned error can't find the container with id c3607f83fff4f6b4a88eefab424976a3534177f1c8e64caed166692257a7151a
Oct 30 04:05:00.742740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:00.742638    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm" event={"ID":"4092798d-6c7a-4dd4-98e1-306a079f0ff6","Type":"ContainerStarted","Data":"58c5537128d13092a5f506f6dfd80c6f68efa2b95e36f901ae2484e0e49895fc"}
Oct 30 04:05:00.742740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:00.742699    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm" event={"ID":"4092798d-6c7a-4dd4-98e1-306a079f0ff6","Type":"ContainerStarted","Data":"c3607f83fff4f6b4a88eefab424976a3534177f1c8e64caed166692257a7151a"}
Oct 30 04:05:00.781468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:00.778699    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm" podStartSLOduration=1.778680636 podStartE2EDuration="1.778680636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:00.775370959 +0000 UTC m=+3729.204260378" watchObservedRunningTime="2026-10-30 04:05:00.778680636 +0000 UTC m=+3729.207570055"
Oct 30 04:05:01.762232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:01.762187    4877 generic.go:356] "Generic (PLEG): container finished" podID="4092798d-6c7a-4dd4-98e1-306a079f0ff6" containerID="58c5537128d13092a5f506f6dfd80c6f68efa2b95e36f901ae2484e0e49895fc" exitCode=0
Oct 30 04:05:01.762790 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:01.762287    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm" event={"ID":"4092798d-6c7a-4dd4-98e1-306a079f0ff6","Type":"ContainerDied","Data":"58c5537128d13092a5f506f6dfd80c6f68efa2b95e36f901ae2484e0e49895fc"}
Oct 30 04:05:02.954730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:02.954703    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:05:03.003692 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.003659    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca\") pod \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") "
Oct 30 04:05:03.003903 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.003718    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca\") pod \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") "
Oct 30 04:05:03.004001 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.003986    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk\" (UniqueName: \"kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk\") pod \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\" (UID: \"4092798d-6c7a-4dd4-98e1-306a079f0ff6\") "
Oct 30 04:05:03.004176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.004149    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca" pod "4092798d-6c7a-4dd4-98e1-306a079f0ff6" (UID: "4092798d-6c7a-4dd4-98e1-306a079f0ff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:05:03.004619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.004594    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:03.006605 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.006574    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk" pod "4092798d-6c7a-4dd4-98e1-306a079f0ff6" (UID: "4092798d-6c7a-4dd4-98e1-306a079f0ff6"). InnerVolumeSpecName "kube-api-access-jkpjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:05:03.007332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.007306    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca" pod "4092798d-6c7a-4dd4-98e1-306a079f0ff6" (UID: "4092798d-6c7a-4dd4-98e1-306a079f0ff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:05:03.106530 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.106495    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4092798d-6c7a-4dd4-98e1-306a079f0ff6-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:03.106530 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.106525    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpjk\" (UniqueName: \"kubernetes.io/projected/4092798d-6c7a-4dd4-98e1-306a079f0ff6-kube-api-access-jkpjk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:03.782295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.782268    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"
Oct 30 04:05:03.782521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.782306    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm" event={"ID":"4092798d-6c7a-4dd4-98e1-306a079f0ff6","Type":"ContainerDied","Data":"c3607f83fff4f6b4a88eefab424976a3534177f1c8e64caed166692257a7151a"}
Oct 30 04:05:03.782521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:03.782356    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3607f83fff4f6b4a88eefab424976a3534177f1c8e64caed166692257a7151a"
Oct 30 04:05:04.119226 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.119185    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"]
Oct 30 04:05:04.119939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.119922    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4092798d-6c7a-4dd4-98e1-306a079f0ff6" containerName="revoke-keys"
Oct 30 04:05:04.119939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.119938    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4092798d-6c7a-4dd4-98e1-306a079f0ff6" containerName="revoke-keys"
Oct 30 04:05:04.120123 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.120110    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4092798d-6c7a-4dd4-98e1-306a079f0ff6" containerName="revoke-keys"
Oct 30 04:05:04.123935 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.123910    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.141828 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.141800    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"]
Oct 30 04:05:04.145677 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.145648    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:05:04.145677 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.145666    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:05:04.235763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.235725    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.235953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.235864    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhw2\" (UniqueName: \"kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.235953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.235940    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.336964 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.336929    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.337159 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.337129    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.337250 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.337231    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhw2\" (UniqueName: \"kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.337760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.337727    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.339691 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.339669    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.354056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.354031    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhw2\" (UniqueName: \"kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2\") pod \"maas-api-revoke-sub-05787b98-8z5gh\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.438717 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.438635    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:04.750887 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.750849    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"]
Oct 30 04:05:04.752660 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:05:04.752629    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad31ddda_b88d_4deb_93c0_7fd22afe99e8.slice/crio-d6068d6a6adcd89505728dbd103edcde16a89480b5c1b395dce96654c0bc1108 WatchSource:0}: Error finding container d6068d6a6adcd89505728dbd103edcde16a89480b5c1b395dce96654c0bc1108: Status 404 returned error can't find the container with id d6068d6a6adcd89505728dbd103edcde16a89480b5c1b395dce96654c0bc1108
Oct 30 04:05:04.795434 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:04.795390    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh" event={"ID":"ad31ddda-b88d-4deb-93c0-7fd22afe99e8","Type":"ContainerStarted","Data":"d6068d6a6adcd89505728dbd103edcde16a89480b5c1b395dce96654c0bc1108"}
Oct 30 04:05:05.810627 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:05.810580    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh" event={"ID":"ad31ddda-b88d-4deb-93c0-7fd22afe99e8","Type":"ContainerStarted","Data":"5da8821a945176fd67dac644e29af7a2a933ab2d51cc0ac13be06f1a41626075"}
Oct 30 04:05:05.859983 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:05.859917    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh" podStartSLOduration=1.859896293 podStartE2EDuration="1.859896293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:05.853857374 +0000 UTC m=+3734.282746793" watchObservedRunningTime="2026-10-30 04:05:05.859896293 +0000 UTC m=+3734.288785712"
Oct 30 04:05:06.823339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:06.823297    4877 generic.go:356] "Generic (PLEG): container finished" podID="ad31ddda-b88d-4deb-93c0-7fd22afe99e8" containerID="5da8821a945176fd67dac644e29af7a2a933ab2d51cc0ac13be06f1a41626075" exitCode=0
Oct 30 04:05:06.823898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:06.823376    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh" event={"ID":"ad31ddda-b88d-4deb-93c0-7fd22afe99e8","Type":"ContainerDied","Data":"5da8821a945176fd67dac644e29af7a2a933ab2d51cc0ac13be06f1a41626075"}
Oct 30 04:05:08.084884 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.084846    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:08.160495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.156983    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2\" (UniqueName: \"kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2\") pod \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") "
Oct 30 04:05:08.160495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.157069    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca\") pod \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") "
Oct 30 04:05:08.160495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.157327    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca\") pod \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\" (UID: \"ad31ddda-b88d-4deb-93c0-7fd22afe99e8\") "
Oct 30 04:05:08.190264 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.190220    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca" pod "ad31ddda-b88d-4deb-93c0-7fd22afe99e8" (UID: "ad31ddda-b88d-4deb-93c0-7fd22afe99e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:05:08.198454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.191323    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:08.226152 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.226113    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca" pod "ad31ddda-b88d-4deb-93c0-7fd22afe99e8" (UID: "ad31ddda-b88d-4deb-93c0-7fd22afe99e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:05:08.244458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.244039    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2" pod "ad31ddda-b88d-4deb-93c0-7fd22afe99e8" (UID: "ad31ddda-b88d-4deb-93c0-7fd22afe99e8"). InnerVolumeSpecName "kube-api-access-fhhw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:05:08.307477 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.304723    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhhw2\" (UniqueName: \"kubernetes.io/projected/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-kube-api-access-fhhw2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:08.307477 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.304765    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad31ddda-b88d-4deb-93c0-7fd22afe99e8-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:08.851466 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.851433    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"
Oct 30 04:05:08.851829 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.851510    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh" event={"ID":"ad31ddda-b88d-4deb-93c0-7fd22afe99e8","Type":"ContainerDied","Data":"d6068d6a6adcd89505728dbd103edcde16a89480b5c1b395dce96654c0bc1108"}
Oct 30 04:05:08.852012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:08.851996    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6068d6a6adcd89505728dbd103edcde16a89480b5c1b395dce96654c0bc1108"
Oct 30 04:05:09.825594 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.825558    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"]
Oct 30 04:05:09.826320 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.826304    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ad31ddda-b88d-4deb-93c0-7fd22afe99e8" containerName="revoke-keys"
Oct 30 04:05:09.826379 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.826321    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad31ddda-b88d-4deb-93c0-7fd22afe99e8" containerName="revoke-keys"
Oct 30 04:05:09.826494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.826484    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad31ddda-b88d-4deb-93c0-7fd22afe99e8" containerName="revoke-keys"
Oct 30 04:05:09.830599 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.830579    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.833788 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.833763    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.833896 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.833818    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.834064 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.834041    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2fn\" (UniqueName: \"kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.848109 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.848075    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"]
Oct 30 04:05:09.852602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.852577    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:05:09.852738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.852627    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:05:09.935451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.935399    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2fn\" (UniqueName: \"kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.935618 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.935527    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.935618 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.935568    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.936255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.936228    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.938115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.938092    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:09.952958 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:09.952931    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2fn\" (UniqueName: \"kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn\") pod \"maas-api-revoke-sub-307dddd6-mbbfz\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:10.145604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:10.145488    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:10.469061 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:10.469016    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"]
Oct 30 04:05:10.469412 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:05:10.469387    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376e4d9a_84f8_4af3_b70e_dfaf9e43dfde.slice/crio-ddfcc1ab5ebcd4833b52d3964d6baf9cb37d2992cc01e1b7136b1ef035f71840 WatchSource:0}: Error finding container ddfcc1ab5ebcd4833b52d3964d6baf9cb37d2992cc01e1b7136b1ef035f71840: Status 404 returned error can't find the container with id ddfcc1ab5ebcd4833b52d3964d6baf9cb37d2992cc01e1b7136b1ef035f71840
Oct 30 04:05:10.976256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:10.976214    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz" event={"ID":"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde","Type":"ContainerStarted","Data":"ddfcc1ab5ebcd4833b52d3964d6baf9cb37d2992cc01e1b7136b1ef035f71840"}
Oct 30 04:05:11.987394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:11.987357    4877 generic.go:356] "Generic (PLEG): container finished" podID="376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" containerID="ab45565b59916bb0600428f0546b6e49477ca1b0dfe92bc39bba8e9636164fa2" exitCode=0
Oct 30 04:05:11.987925 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:11.987467    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz" event={"ID":"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde","Type":"ContainerDied","Data":"ab45565b59916bb0600428f0546b6e49477ca1b0dfe92bc39bba8e9636164fa2"}
Oct 30 04:05:13.190255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.190228    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:13.228818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.228758    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca\") pod \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") "
Oct 30 04:05:13.228818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.228816    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca\") pod \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") "
Oct 30 04:05:13.229070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.228953    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn\" (UniqueName: \"kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn\") pod \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\" (UID: \"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde\") "
Oct 30 04:05:13.229159 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.229133    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca" pod "376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" (UID: "376e4d9a-84f8-4af3-b70e-dfaf9e43dfde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:05:13.229874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.229816    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:13.231513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.231485    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn" pod "376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" (UID: "376e4d9a-84f8-4af3-b70e-dfaf9e43dfde"). InnerVolumeSpecName "kube-api-access-fb2fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:05:13.231728 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.231706    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca" pod "376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" (UID: "376e4d9a-84f8-4af3-b70e-dfaf9e43dfde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:05:13.331763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.331728    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:13.331763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:13.331756    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2fn\" (UniqueName: \"kubernetes.io/projected/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde-kube-api-access-fb2fn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:14.008188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.008101    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"
Oct 30 04:05:14.008342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.008102    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz" event={"ID":"376e4d9a-84f8-4af3-b70e-dfaf9e43dfde","Type":"ContainerDied","Data":"ddfcc1ab5ebcd4833b52d3964d6baf9cb37d2992cc01e1b7136b1ef035f71840"}
Oct 30 04:05:14.008342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.008262    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddfcc1ab5ebcd4833b52d3964d6baf9cb37d2992cc01e1b7136b1ef035f71840"
Oct 30 04:05:14.707933 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.707900    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"]
Oct 30 04:05:14.709058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.709033    4877 cpu_manager.go:414] "containerMap: removing container" podUID="376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" containerName="revoke-keys"
Oct 30 04:05:14.709058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.709057    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" containerName="revoke-keys"
Oct 30 04:05:14.709302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.709287    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" containerName="revoke-keys"
Oct 30 04:05:14.713980 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.713959    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.729260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.729233    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"]
Oct 30 04:05:14.731734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.731708    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:05:14.731835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.731781    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:05:14.852362 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.852323    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fwp\" (UniqueName: \"kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.852593 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.852375    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.852593 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.852566    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.954193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.954158    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fwp\" (UniqueName: \"kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.954363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.954200    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.954363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.954288    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.954916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.954893    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.956754 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.956729    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:14.970998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:14.970940    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fwp\" (UniqueName: \"kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp\") pod \"maas-api-revoke-sub-5cf1edc9-6kd6j\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:15.033686 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:15.033652    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:15.342235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:15.342204    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"]
Oct 30 04:05:15.344148 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:05:15.344118    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fe115c_36f5_4e69_b643_09cf657a36c8.slice/crio-7bcd5077372fa91f2a9543eaab550b7dcefbe76ed7a30672130008007f5aa54f WatchSource:0}: Error finding container 7bcd5077372fa91f2a9543eaab550b7dcefbe76ed7a30672130008007f5aa54f: Status 404 returned error can't find the container with id 7bcd5077372fa91f2a9543eaab550b7dcefbe76ed7a30672130008007f5aa54f
Oct 30 04:05:16.031133 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:16.031099    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j" event={"ID":"c1fe115c-36f5-4e69-b643-09cf657a36c8","Type":"ContainerStarted","Data":"fd64196683b0ebe6139b781dcb85301ba0db56c9147e1afa855e8ae517c100a6"}
Oct 30 04:05:16.031545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:16.031159    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j" event={"ID":"c1fe115c-36f5-4e69-b643-09cf657a36c8","Type":"ContainerStarted","Data":"7bcd5077372fa91f2a9543eaab550b7dcefbe76ed7a30672130008007f5aa54f"}
Oct 30 04:05:16.064525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:16.064472    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j" podStartSLOduration=2.064458873 podStartE2EDuration="2.064458873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:16.0637287 +0000 UTC m=+3744.492618120" watchObservedRunningTime="2026-10-30 04:05:16.064458873 +0000 UTC m=+3744.493348284"
Oct 30 04:05:16.231018 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:05:16.230977    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:05:17.054641 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:17.054604    4877 generic.go:356] "Generic (PLEG): container finished" podID="c1fe115c-36f5-4e69-b643-09cf657a36c8" containerID="fd64196683b0ebe6139b781dcb85301ba0db56c9147e1afa855e8ae517c100a6" exitCode=0
Oct 30 04:05:17.055148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:17.054890    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j" event={"ID":"c1fe115c-36f5-4e69-b643-09cf657a36c8","Type":"ContainerDied","Data":"fd64196683b0ebe6139b781dcb85301ba0db56c9147e1afa855e8ae517c100a6"}
Oct 30 04:05:18.279470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.279445    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:18.432394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.430621    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca\") pod \"c1fe115c-36f5-4e69-b643-09cf657a36c8\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") "
Oct 30 04:05:18.432957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.432924    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp\" (UniqueName: \"kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp\") pod \"c1fe115c-36f5-4e69-b643-09cf657a36c8\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") "
Oct 30 04:05:18.433253 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.433235    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca\") pod \"c1fe115c-36f5-4e69-b643-09cf657a36c8\" (UID: \"c1fe115c-36f5-4e69-b643-09cf657a36c8\") "
Oct 30 04:05:18.434497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.434470    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca" pod "c1fe115c-36f5-4e69-b643-09cf657a36c8" (UID: "c1fe115c-36f5-4e69-b643-09cf657a36c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:05:18.468947 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.468591    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca" pod "c1fe115c-36f5-4e69-b643-09cf657a36c8" (UID: "c1fe115c-36f5-4e69-b643-09cf657a36c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:05:18.474008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.473959    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp" pod "c1fe115c-36f5-4e69-b643-09cf657a36c8" (UID: "c1fe115c-36f5-4e69-b643-09cf657a36c8"). InnerVolumeSpecName "kube-api-access-r7fwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:05:18.535820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.535710    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fwp\" (UniqueName: \"kubernetes.io/projected/c1fe115c-36f5-4e69-b643-09cf657a36c8-kube-api-access-r7fwp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:18.535820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.535762    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:18.535820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:18.535781    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fe115c-36f5-4e69-b643-09cf657a36c8-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:05:19.111100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:19.111069    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"
Oct 30 04:05:19.111272 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:19.111096    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j" event={"ID":"c1fe115c-36f5-4e69-b643-09cf657a36c8","Type":"ContainerDied","Data":"7bcd5077372fa91f2a9543eaab550b7dcefbe76ed7a30672130008007f5aa54f"}
Oct 30 04:05:19.111272 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:19.111156    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bcd5077372fa91f2a9543eaab550b7dcefbe76ed7a30672130008007f5aa54f"
Oct 30 04:05:30.028622 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:30.028582    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"]
Oct 30 04:05:30.033411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:30.033384    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0012b598-hcg5v"]
Oct 30 04:05:31.701355 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:31.701320    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75aefa7e-f7a5-485d-b74d-57d50f109b66" path="/var/lib/kubelet/pods/75aefa7e-f7a5-485d-b74d-57d50f109b66/volumes"
Oct 30 04:05:36.688894 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.688857    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"]
Oct 30 04:05:36.689647 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.689627    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c1fe115c-36f5-4e69-b643-09cf657a36c8" containerName="revoke-keys"
Oct 30 04:05:36.689647 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.689646    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fe115c-36f5-4e69-b643-09cf657a36c8" containerName="revoke-keys"
Oct 30 04:05:36.689805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.689775    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fe115c-36f5-4e69-b643-09cf657a36c8" containerName="revoke-keys"
Oct 30 04:05:36.694231 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.694199    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.727604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.727565    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-dd81fd-openshift-default-dockercfg-g9rm4\""
Oct 30 04:05:36.734731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.734666    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"]
Oct 30 04:05:36.872097 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.872055    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.872407 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.872381    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9f4\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.872555 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.872538    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.872659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.872646    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.872785 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.872772    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.873072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.873057    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.873282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.873269    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.894444 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.873538    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.894736 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.894706    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.998968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.998868    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:36.999704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:36.999677    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.000381 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.000360    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.001436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.001402    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.001872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.001850    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.002308 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.001783    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.002470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.001237    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.002470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.002206    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.002470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.000258    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.002730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.002701    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.003198 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.003177    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9f4\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.009312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.009288    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.009849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.009829    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.010531 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.009773    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.037570 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.037449    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.046384 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.046306    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.047052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.046985    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.057162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.057083    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9f4\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.325593 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.325556    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:37.657620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.657575    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"]
Oct 30 04:05:37.658492 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:05:37.658462    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf435747d_c138_430d_9a05_c3789c910abf.slice/crio-878032dc4b42948dd5ca232df14bdb62239eaa500144591e5cb54b3543397511 WatchSource:0}: Error finding container 878032dc4b42948dd5ca232df14bdb62239eaa500144591e5cb54b3543397511: Status 404 returned error can't find the container with id 878032dc4b42948dd5ca232df14bdb62239eaa500144591e5cb54b3543397511
Oct 30 04:05:37.661352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.661322    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:05:37.661416 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.661387    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:05:37.661491 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:37.661473    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:05:38.287758 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:38.287717    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" event={"ID":"f435747d-c138-430d-9a05-c3789c910abf","Type":"ContainerStarted","Data":"963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"}
Oct 30 04:05:38.288248 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:38.287823    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" event={"ID":"f435747d-c138-430d-9a05-c3789c910abf","Type":"ContainerStarted","Data":"878032dc4b42948dd5ca232df14bdb62239eaa500144591e5cb54b3543397511"}
Oct 30 04:05:38.325748 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:38.325712    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:38.330669 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:38.330606    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podStartSLOduration=2.330587224 podStartE2EDuration="2.330587224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:38.325209795 +0000 UTC m=+3766.754099213" watchObservedRunningTime="2026-10-30 04:05:38.330587224 +0000 UTC m=+3766.759476641"
Oct 30 04:05:38.335161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:38.335136    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:39.296911 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:39.296870    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:39.298169 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:39.298148    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:05:40.813619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:40.813579    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"]
Oct 30 04:05:40.848226 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:40.848144    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"]
Oct 30 04:05:45.814721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:45.814656    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerName="payload-processing" containerID="cri-o://c030d509fd8717ffdba4ec749475b890d9e66d4e53ace1fdedc535e28b3571b0" gracePeriod=25
Oct 30 04:05:45.849071 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:45.849021    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerName="payload-pre-processing" containerID="cri-o://fd65c96a5c4fd99438bf356d4587ce1b99e32c64e513e9c26a49b854bc795142" gracePeriod=25
Oct 30 04:05:46.230351 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:05:46.230267    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:05:49.224453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:49.213527    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:05:50.216808 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:50.216748    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:05:57.163696 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:57.163660    4877 scope.go:122] "RemoveContainer" containerID="53006d6cfe639bd060f3c8ec5d111c6d8367056dd93e0b9959945779afb9e521"
Oct 30 04:05:57.176162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:57.176142    4877 scope.go:122] "RemoveContainer" containerID="f9b3ab5a69cc73b7de8fccf8cb79a5ad728e4ddbe9f3c5e295e0f6c85101ae04"
Oct 30 04:05:57.942633 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:57.942594    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"]
Oct 30 04:05:57.942890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:57.942859    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy" containerID="cri-o://42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c" gracePeriod=30
Oct 30 04:05:59.188045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:05:59.188002    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:06:00.196001 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:00.195961    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:06:03.276289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.276262    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 04:06:03.355077 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355044    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.355253 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355143    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.355253 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355219    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355311    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355348    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355364    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355376    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355506    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355579    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355585    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355716    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert\") pod \"300ee1d9-af02-4b23-93e4-d90143518f26\" (UID: \"300ee1d9-af02-4b23-93e4-d90143518f26\") "
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.355844    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.356022    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.356597    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.356619    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.356631    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.356635    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:03.358581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.356646    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.366317 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.366192    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:03.366317 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.366250    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:06:03.366317 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.366289    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "kube-api-access-frjtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:03.366658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.366635    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token" pod "300ee1d9-af02-4b23-93e4-d90143518f26" (UID: "300ee1d9-af02-4b23-93e4-d90143518f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:03.457852 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.457808    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.457852 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.457844    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjtf\" (UniqueName: \"kubernetes.io/projected/300ee1d9-af02-4b23-93e4-d90143518f26-kube-api-access-frjtf\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.458070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.457862    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300ee1d9-af02-4b23-93e4-d90143518f26-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.458070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.457875    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300ee1d9-af02-4b23-93e4-d90143518f26-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.458070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.457887    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300ee1d9-af02-4b23-93e4-d90143518f26-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:03.999247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.999157    4877 generic.go:356] "Generic (PLEG): container finished" podID="300ee1d9-af02-4b23-93e4-d90143518f26" containerID="42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c" exitCode=0
Oct 30 04:06:03.999247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.999228    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"
Oct 30 04:06:03.999558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.999242    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" event={"ID":"300ee1d9-af02-4b23-93e4-d90143518f26","Type":"ContainerDied","Data":"42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c"}
Oct 30 04:06:03.999558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.999304    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62" event={"ID":"300ee1d9-af02-4b23-93e4-d90143518f26","Type":"ContainerDied","Data":"cf37b87934e4e76d423fbebd5a0b179d3010e217a23e8f40360c42d08ce6ecfb"}
Oct 30 04:06:03.999558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:03.999340    4877 scope.go:122] "RemoveContainer" containerID="42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c"
Oct 30 04:06:04.010833 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.010812    4877 scope.go:122] "RemoveContainer" containerID="ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"
Oct 30 04:06:04.022630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.022609    4877 scope.go:122] "RemoveContainer" containerID="42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c"
Oct 30 04:06:04.022907 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:04.022886    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c\": container with ID starting with 42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c not found: ID does not exist" containerID="42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c"
Oct 30 04:06:04.023002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.022916    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c"} err="failed to get container status \"42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c\": rpc error: code = NotFound desc = could not find container \"42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c\": container with ID starting with 42ef383b1397d05adc83216b23bd699eea64b27e14e4f7d0e7f089c25c52b79c not found: ID does not exist"
Oct 30 04:06:04.023002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.022933    4877 scope.go:122] "RemoveContainer" containerID="ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"
Oct 30 04:06:04.023251 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:04.023231    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6\": container with ID starting with ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6 not found: ID does not exist" containerID="ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"
Oct 30 04:06:04.023313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.023260    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6"} err="failed to get container status \"ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6\": rpc error: code = NotFound desc = could not find container \"ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6\": container with ID starting with ea5164f30499e054bc135e677357a0d3d0dd3d4e985778a28b846fed1f82daf6 not found: ID does not exist"
Oct 30 04:06:04.059886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.059861    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"]
Oct 30 04:06:04.064613 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:04.064591    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9d8692-openshift-default-6b89b6489-9gn62"]
Oct 30 04:06:05.700405 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:05.700372    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" path="/var/lib/kubelet/pods/300ee1d9-af02-4b23-93e4-d90143518f26/volumes"
Oct 30 04:06:06.038479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.038209    4877 generic.go:356] "Generic (PLEG): container finished" podID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerID="c030d509fd8717ffdba4ec749475b890d9e66d4e53ace1fdedc535e28b3571b0" exitCode=0
Oct 30 04:06:06.038479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.038371    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" event={"ID":"4ad02199-2cdc-4e9a-98cf-8200baf66843","Type":"ContainerDied","Data":"c030d509fd8717ffdba4ec749475b890d9e66d4e53ace1fdedc535e28b3571b0"}
Oct 30 04:06:06.044724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.044518    4877 generic.go:356] "Generic (PLEG): container finished" podID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerID="fd65c96a5c4fd99438bf356d4587ce1b99e32c64e513e9c26a49b854bc795142" exitCode=0
Oct 30 04:06:06.044724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.044651    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" event={"ID":"bd7a9816-26a9-46ca-b754-493cf1e031d8","Type":"ContainerDied","Data":"fd65c96a5c4fd99438bf356d4587ce1b99e32c64e513e9c26a49b854bc795142"}
Oct 30 04:06:06.178203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.178173    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 04:06:06.214560 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.214503    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 04:06:06.328214 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.328171    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume\") pod \"bd7a9816-26a9-46ca-b754-493cf1e031d8\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") "
Oct 30 04:06:06.328389 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.328272    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume\") pod \"4ad02199-2cdc-4e9a-98cf-8200baf66843\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") "
Oct 30 04:06:06.328510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.328488    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h\" (UniqueName: \"kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h\") pod \"bd7a9816-26a9-46ca-b754-493cf1e031d8\" (UID: \"bd7a9816-26a9-46ca-b754-493cf1e031d8\") "
Oct 30 04:06:06.328577 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.328521    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l\" (UniqueName: \"kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l\") pod \"4ad02199-2cdc-4e9a-98cf-8200baf66843\" (UID: \"4ad02199-2cdc-4e9a-98cf-8200baf66843\") "
Oct 30 04:06:06.328648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.328596    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume" pod "bd7a9816-26a9-46ca-b754-493cf1e031d8" (UID: "bd7a9816-26a9-46ca-b754-493cf1e031d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:06.328718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.328680    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume" pod "4ad02199-2cdc-4e9a-98cf-8200baf66843" (UID: "4ad02199-2cdc-4e9a-98cf-8200baf66843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:06.329067 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.329046    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad02199-2cdc-4e9a-98cf-8200baf66843-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:06.329067 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.329064    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7a9816-26a9-46ca-b754-493cf1e031d8-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:06.330692 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.330671    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h" pod "bd7a9816-26a9-46ca-b754-493cf1e031d8" (UID: "bd7a9816-26a9-46ca-b754-493cf1e031d8"). InnerVolumeSpecName "kube-api-access-n2n9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:06.330827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.330807    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l" pod "4ad02199-2cdc-4e9a-98cf-8200baf66843" (UID: "4ad02199-2cdc-4e9a-98cf-8200baf66843"). InnerVolumeSpecName "kube-api-access-jg87l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:06.430795 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.430763    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2n9h\" (UniqueName: \"kubernetes.io/projected/bd7a9816-26a9-46ca-b754-493cf1e031d8-kube-api-access-n2n9h\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:06.430795 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:06.430792    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg87l\" (UniqueName: \"kubernetes.io/projected/4ad02199-2cdc-4e9a-98cf-8200baf66843-kube-api-access-jg87l\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:07.056702 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.056657    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"
Oct 30 04:06:07.057176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.056688    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l" event={"ID":"4ad02199-2cdc-4e9a-98cf-8200baf66843","Type":"ContainerDied","Data":"95f8d0e3822ea21b8aa87c3e49ac5b4f791135756f7f9d421a31508da8a28622"}
Oct 30 04:06:07.057176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.056776    4877 scope.go:122] "RemoveContainer" containerID="c030d509fd8717ffdba4ec749475b890d9e66d4e53ace1fdedc535e28b3571b0"
Oct 30 04:06:07.076841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.076550    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx" event={"ID":"bd7a9816-26a9-46ca-b754-493cf1e031d8","Type":"ContainerDied","Data":"bffc8f9145c858bab3d25c82bd734f1db8949d9da7d1f9a2f47f851e80f7995a"}
Oct 30 04:06:07.076841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.076673    4877 scope.go:122] "RemoveContainer" containerID="fd65c96a5c4fd99438bf356d4587ce1b99e32c64e513e9c26a49b854bc795142"
Oct 30 04:06:07.076841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.076684    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"
Oct 30 04:06:07.127358 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.127314    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"]
Oct 30 04:06:07.135858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.135826    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-689bcf-59fdddd88d-xp56l"]
Oct 30 04:06:07.167803 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.165823    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"]
Oct 30 04:06:07.172938 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.172915    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-689bcf-5dbc7c7775-gvzcx"]
Oct 30 04:06:07.741452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.739120    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" path="/var/lib/kubelet/pods/4ad02199-2cdc-4e9a-98cf-8200baf66843/volumes"
Oct 30 04:06:07.741452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:07.739668    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" path="/var/lib/kubelet/pods/bd7a9816-26a9-46ca-b754-493cf1e031d8/volumes"
Oct 30 04:06:10.028207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:10.028174    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"]
Oct 30 04:06:10.037411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:10.037382    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d88702-zrhfq"]
Oct 30 04:06:11.700541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:11.700510    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78b5171-a071-456e-aac4-5834e5b021f3" path="/var/lib/kubelet/pods/c78b5171-a071-456e-aac4-5834e5b021f3/volumes"
Oct 30 04:06:14.779526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:14.779489    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"]
Oct 30 04:06:14.780345 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:14.780310    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy" containerID="cri-o://ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951" gracePeriod=30
Oct 30 04:06:16.231806 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:16.231761    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:06:17.083347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:17.083304    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"]
Oct 30 04:06:17.084394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:17.083609    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" containerID="cri-o://b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254" gracePeriod=30
Oct 30 04:06:17.538597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:17.538561    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:06:17.538999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:17.538628    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:06:20.095058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.095027    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 04:06:20.184810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.184770    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.184895    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.184952    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185004    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185072    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185113    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:20.185230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185128    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185409 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185223    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185409 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185302    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.185552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.185475    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs\") pod \"d818f979-ebd7-448b-b761-3aa7d712a85c\" (UID: \"d818f979-ebd7-448b-b761-3aa7d712a85c\") "
Oct 30 04:06:20.186125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.186098    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.186358 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.186336    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:20.186811 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.186735    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:20.187131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.187066    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:20.187486 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.187411    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:20.187942 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.187904    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:20.189328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.189290    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:20.189846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.189821    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:06:20.190142 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.190118    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x" pod "d818f979-ebd7-448b-b761-3aa7d712a85c" (UID: "d818f979-ebd7-448b-b761-3aa7d712a85c"). InnerVolumeSpecName "kube-api-access-9m97x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:20.288033 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288000    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d818f979-ebd7-448b-b761-3aa7d712a85c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288033 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288027    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288033 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288037    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288047    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288056    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288064    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288072    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m97x\" (UniqueName: \"kubernetes.io/projected/d818f979-ebd7-448b-b761-3aa7d712a85c-kube-api-access-9m97x\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.288280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.288080    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818f979-ebd7-448b-b761-3aa7d712a85c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:20.357197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.357119    4877 generic.go:356] "Generic (PLEG): container finished" podID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerID="ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951" exitCode=0
Oct 30 04:06:20.357365 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.357331    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" event={"ID":"d818f979-ebd7-448b-b761-3aa7d712a85c","Type":"ContainerDied","Data":"ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951"}
Oct 30 04:06:20.357455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.357389    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw" event={"ID":"d818f979-ebd7-448b-b761-3aa7d712a85c","Type":"ContainerDied","Data":"01161ccf85e2dbb43a80a383f2b95e4f486f8cd681d7d28851bad402118c09af"}
Oct 30 04:06:20.357455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.357438    4877 scope.go:122] "RemoveContainer" containerID="ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951"
Oct 30 04:06:20.357619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.357602    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"
Oct 30 04:06:20.377964 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.377941    4877 scope.go:122] "RemoveContainer" containerID="a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"
Oct 30 04:06:20.389313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.389282    4877 scope.go:122] "RemoveContainer" containerID="ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951"
Oct 30 04:06:20.389616 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:20.389590    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951\": container with ID starting with ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951 not found: ID does not exist" containerID="ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951"
Oct 30 04:06:20.389703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.389622    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951"} err="failed to get container status \"ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951\": rpc error: code = NotFound desc = could not find container \"ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951\": container with ID starting with ce03832effbff860f249ad1500039f38654fcf77ba74492af155b4e8974ef951 not found: ID does not exist"
Oct 30 04:06:20.389703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.389639    4877 scope.go:122] "RemoveContainer" containerID="a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"
Oct 30 04:06:20.389923 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:20.389898    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5\": container with ID starting with a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5 not found: ID does not exist" containerID="a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"
Oct 30 04:06:20.390015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.389930    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5"} err="failed to get container status \"a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5\": rpc error: code = NotFound desc = could not find container \"a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5\": container with ID starting with a68dbb96463c535ca06bef68a1d1513b3086d53e75541c48ae2acb47e790adf5 not found: ID does not exist"
Oct 30 04:06:20.412212 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.412185    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"]
Oct 30 04:06:20.422465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.422439    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-93b821-openshift-default-6bfd44fd77-sl2lw"]
Oct 30 04:06:20.959303 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:20.959259    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"]
Oct 30 04:06:21.037200 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:21.033365    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"]
Oct 30 04:06:21.037534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:21.037505    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f3a1fd-hkjv6"]
Oct 30 04:06:21.729447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:21.729008    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8dd85c6-fd63-46d7-982b-f8f30b926ba8" path="/var/lib/kubelet/pods/b8dd85c6-fd63-46d7-982b-f8f30b926ba8/volumes"
Oct 30 04:06:21.729963 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:21.729702    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" path="/var/lib/kubelet/pods/d818f979-ebd7-448b-b761-3aa7d712a85c/volumes"
Oct 30 04:06:22.366146 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.366085    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 04:06:22.387059 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.387022    4877 generic.go:356] "Generic (PLEG): container finished" podID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerID="b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254" exitCode=0
Oct 30 04:06:22.387208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.387069    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" event={"ID":"dd1b44f8-0fcb-4f16-85e1-87698900b711","Type":"ContainerDied","Data":"b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254"}
Oct 30 04:06:22.387208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.387088    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"
Oct 30 04:06:22.387208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.387126    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6" event={"ID":"dd1b44f8-0fcb-4f16-85e1-87698900b711","Type":"ContainerDied","Data":"a24d8c8fcd394f9c0de0b5378ff2b3bc273c733cab22cbb8011986b3f6702dae"}
Oct 30 04:06:22.387208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.387157    4877 scope.go:122] "RemoveContainer" containerID="b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254"
Oct 30 04:06:22.404771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.404165    4877 scope.go:122] "RemoveContainer" containerID="7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"
Oct 30 04:06:22.422859 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.422390    4877 scope.go:122] "RemoveContainer" containerID="b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254"
Oct 30 04:06:22.428597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.428567    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.428730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.428714    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.428817 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.428798    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.428923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.428906    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.429003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.428958    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.429112 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.429099    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.429176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.429144    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.429262 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.429242    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.429342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.429309    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert\") pod \"dd1b44f8-0fcb-4f16-85e1-87698900b711\" (UID: \"dd1b44f8-0fcb-4f16-85e1-87698900b711\") "
Oct 30 04:06:22.430474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.429866    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:22.430474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430005    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.430474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430095    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:22.430474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430203    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:22.430474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430382    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:22.430816 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:22.430502    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254\": container with ID starting with b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254 not found: ID does not exist" containerID="b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254"
Oct 30 04:06:22.430816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430558    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254"} err="failed to get container status \"b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254\": rpc error: code = NotFound desc = could not find container \"b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254\": container with ID starting with b25925a48e7e84b90d1494af6718e381173cd065ec30790fbbd316d66272d254 not found: ID does not exist"
Oct 30 04:06:22.430816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430584    4877 scope.go:122] "RemoveContainer" containerID="7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"
Oct 30 04:06:22.430816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.430623    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:22.431068 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:22.431032    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083\": container with ID starting with 7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083 not found: ID does not exist" containerID="7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"
Oct 30 04:06:22.431184 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.431072    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083"} err="failed to get container status \"7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083\": rpc error: code = NotFound desc = could not find container \"7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083\": container with ID starting with 7c23fd37fc1c6ffa48eb837d72dc439d4747d4e7f535623921867e7150f0e083 not found: ID does not exist"
Oct 30 04:06:22.431945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.431916    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:06:22.432450 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.432414    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:22.432824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.432800    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:22.434123 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.434098    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s" pod "dd1b44f8-0fcb-4f16-85e1-87698900b711" (UID: "dd1b44f8-0fcb-4f16-85e1-87698900b711"). InnerVolumeSpecName "kube-api-access-6f84s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:22.531093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531059    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531091    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531100    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531110    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531119    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1b44f8-0fcb-4f16-85e1-87698900b711-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531126    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1b44f8-0fcb-4f16-85e1-87698900b711-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531135    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1b44f8-0fcb-4f16-85e1-87698900b711-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.531321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.531142    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f84s\" (UniqueName: \"kubernetes.io/projected/dd1b44f8-0fcb-4f16-85e1-87698900b711-kube-api-access-6f84s\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:22.731842 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.731791    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"]
Oct 30 04:06:22.736406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:22.736381    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-689bcf-openshift-default-7f7f54598f-bcpr6"]
Oct 30 04:06:23.700872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:23.700836    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" path="/var/lib/kubelet/pods/dd1b44f8-0fcb-4f16-85e1-87698900b711/volumes"
Oct 30 04:06:25.959701 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:25.959657    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerName="payload-pre-processing" containerID="cri-o://e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62" gracePeriod=25
Oct 30 04:06:26.766919 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.766885    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"]
Oct 30 04:06:26.767683 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767666    4877 cpu_manager.go:414] "containerMap: removing container" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy"
Oct 30 04:06:26.767737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767684    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy"
Oct 30 04:06:26.767737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767693    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy"
Oct 30 04:06:26.767737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767698    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy"
Oct 30 04:06:26.767737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767723    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerName="payload-pre-processing"
Oct 30 04:06:26.767737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767729    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerName="payload-pre-processing"
Oct 30 04:06:26.767737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767738    4877 cpu_manager.go:414] "containerMap: removing container" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767743    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767757    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerName="payload-processing"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767762    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerName="payload-processing"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767772    4877 cpu_manager.go:414] "containerMap: removing container" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767776    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767787    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767792    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767804    4877 cpu_manager.go:414] "containerMap: removing container" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767809    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767899    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7a9816-26a9-46ca-b754-493cf1e031d8" containerName="payload-pre-processing"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767912    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767919    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad02199-2cdc-4e9a-98cf-8200baf66843" containerName="payload-processing"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767926    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="300ee1d9-af02-4b23-93e4-d90143518f26" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767931    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy"
Oct 30 04:06:26.767931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767938    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy"
Oct 30 04:06:26.768373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.767946    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1b44f8-0fcb-4f16-85e1-87698900b711" containerName="istio-proxy"
Oct 30 04:06:26.773101 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.773084    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:26.786751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.786728    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:06:26.796706 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.796677    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:06:26.803326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.803298    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"]
Oct 30 04:06:26.907869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.907827    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:26.908049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.907966    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:26.908137 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:26.908106    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2qs\" (UniqueName: \"kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.009539 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.009501    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.010016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.009631    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2qs\" (UniqueName: \"kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.010016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.009768    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.010468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.010447    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.012060 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.012035    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.028436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.028409    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2qs\" (UniqueName: \"kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs\") pod \"maas-api-revoke-sub-6cfa8300-dxb5z\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.089569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.089534    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:27.207541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.207501    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:06:27.402520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.402384    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"]
Oct 30 04:06:27.404598 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:06:27.404562    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0f14d5_80c7_466e_b0e8_cb52a8f5bc7e.slice/crio-d3403ce138aa3fe757efe654aeb04f69a9585135142529a0e467b554643df479 WatchSource:0}: Error finding container d3403ce138aa3fe757efe654aeb04f69a9585135142529a0e467b554643df479: Status 404 returned error can't find the container with id d3403ce138aa3fe757efe654aeb04f69a9585135142529a0e467b554643df479
Oct 30 04:06:27.480492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:27.480459    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z" event={"ID":"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e","Type":"ContainerStarted","Data":"d3403ce138aa3fe757efe654aeb04f69a9585135142529a0e467b554643df479"}
Oct 30 04:06:28.516502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:28.516463    4877 generic.go:356] "Generic (PLEG): container finished" podID="0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" containerID="c977e3709945dd868c81df9a60d6bee2246692f64dd1311b4372bd690d2e6f91" exitCode=0
Oct 30 04:06:28.516843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:28.516722    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z" event={"ID":"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e","Type":"ContainerDied","Data":"c977e3709945dd868c81df9a60d6bee2246692f64dd1311b4372bd690d2e6f91"}
Oct 30 04:06:29.706307 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.706270    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:29.761066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.761028    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs\" (UniqueName: \"kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs\") pod \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") "
Oct 30 04:06:29.761265 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.761218    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca\") pod \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") "
Oct 30 04:06:29.761338 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.761296    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca\") pod \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\" (UID: \"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e\") "
Oct 30 04:06:29.761721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.761690    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca" pod "0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" (UID: "0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:29.762481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.762454    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:29.763655 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.763626    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca" pod "0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" (UID: "0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:29.763735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.763677    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs" pod "0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" (UID: "0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e"). InnerVolumeSpecName "kube-api-access-9p2qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:29.864267 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.864186    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p2qs\" (UniqueName: \"kubernetes.io/projected/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-kube-api-access-9p2qs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:29.864267 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:29.864225    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:30.537291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:30.537255    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z" event={"ID":"0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e","Type":"ContainerDied","Data":"d3403ce138aa3fe757efe654aeb04f69a9585135142529a0e467b554643df479"}
Oct 30 04:06:30.537291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:30.537277    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"
Oct 30 04:06:30.537565 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:30.537305    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3403ce138aa3fe757efe654aeb04f69a9585135142529a0e467b554643df479"
Oct 30 04:06:32.400740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:32.400104    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"]
Oct 30 04:06:32.401240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:32.400745    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy" containerID="cri-o://483a7d4e81846b6a74ea991ba3246266879a607a699318071cedfb2b4ec59919" gracePeriod=30
Oct 30 04:06:37.097514 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.097417    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"]
Oct 30 04:06:37.098145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.098128    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" containerName="revoke-keys"
Oct 30 04:06:37.098194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.098147    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" containerName="revoke-keys"
Oct 30 04:06:37.098282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.098273    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d818f979-ebd7-448b-b761-3aa7d712a85c" containerName="istio-proxy"
Oct 30 04:06:37.098318 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.098291    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" containerName="revoke-keys"
Oct 30 04:06:37.102571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.102552    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.120859 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.120825    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"]
Oct 30 04:06:37.126173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.126149    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:06:37.126270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.126180    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:06:37.187308 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.187279    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.187489 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.187315    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv5x\" (UniqueName: \"kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.187540 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.187507    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.207017 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.206973    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:06:37.288340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.288295    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.288503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.288471    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.288503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.288491    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv5x\" (UniqueName: \"kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.289111 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.289087    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.290691 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.290667    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.305768 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.305743    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv5x\" (UniqueName: \"kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x\") pod \"maas-api-revoke-sub-95895851-ml9hr\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.416490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.416381    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:37.665518 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.665481    4877 generic.go:356] "Generic (PLEG): container finished" podID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerID="483a7d4e81846b6a74ea991ba3246266879a607a699318071cedfb2b4ec59919" exitCode=0
Oct 30 04:06:37.665716 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.665700    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" event={"ID":"a07f1aad-625a-4cc5-b41c-6361b464df90","Type":"ContainerDied","Data":"483a7d4e81846b6a74ea991ba3246266879a607a699318071cedfb2b4ec59919"}
Oct 30 04:06:37.665799 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.665759    4877 scope.go:122] "RemoveContainer" containerID="aa321d45e0da2829935b3381269144f914b2d0ca8bf9e8f4f734c683e0786c91"
Oct 30 04:06:37.739351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.739330    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 04:06:37.740576 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.740554    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"]
Oct 30 04:06:37.741976 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:06:37.741952    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a87329d_e60b_404a_95d2_b32e6fc6001a.slice/crio-8d022c5c641c0fe2b519689738add86a8f8f1f965c6a642dc5115bc83add86c4 WatchSource:0}: Error finding container 8d022c5c641c0fe2b519689738add86a8f8f1f965c6a642dc5115bc83add86c4: Status 404 returned error can't find the container with id 8d022c5c641c0fe2b519689738add86a8f8f1f965c6a642dc5115bc83add86c4
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.800781    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.800869    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.800902    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.800948    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.801019    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.801137    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.801202    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.801323    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.801936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.801369    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token\") pod \"a07f1aad-625a-4cc5-b41c-6361b464df90\" (UID: \"a07f1aad-625a-4cc5-b41c-6361b464df90\") "
Oct 30 04:06:37.804384 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.804359    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:37.804684 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.804554    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:37.804891 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.804865    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:37.811010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.810977    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:37.811286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.811263    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:37.812856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.812554    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:37.818004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.817976    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:37.818227 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.818200    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "kube-api-access-g6f6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:37.819529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.819503    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo" pod "a07f1aad-625a-4cc5-b41c-6361b464df90" (UID: "a07f1aad-625a-4cc5-b41c-6361b464df90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:06:37.826744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.826721    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"]
Oct 30 04:06:37.826933 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.826907    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy" containerID="cri-o://f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef" gracePeriod=30
Oct 30 04:06:37.905402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905361    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905396    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905411    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905441    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905456    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905471    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f1aad-625a-4cc5-b41c-6361b464df90-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905485    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6f6n\" (UniqueName: \"kubernetes.io/projected/a07f1aad-625a-4cc5-b41c-6361b464df90-kube-api-access-g6f6n\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905500    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:37.905712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:37.905515    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f1aad-625a-4cc5-b41c-6361b464df90-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:38.678275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.678138    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr" event={"ID":"6a87329d-e60b-404a-95d2-b32e6fc6001a","Type":"ContainerStarted","Data":"ead0d30745c6cf89b51c64fe73a207a333e27a599b57601fe0ccf058c48fa7ca"}
Oct 30 04:06:38.678861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.678282    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr" event={"ID":"6a87329d-e60b-404a-95d2-b32e6fc6001a","Type":"ContainerStarted","Data":"8d022c5c641c0fe2b519689738add86a8f8f1f965c6a642dc5115bc83add86c4"}
Oct 30 04:06:38.680412 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.680378    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4" event={"ID":"a07f1aad-625a-4cc5-b41c-6361b464df90","Type":"ContainerDied","Data":"3ff6928c70e950b356fff022e27a73ba9c8066a4ea50366a17f2672c0d1665d2"}
Oct 30 04:06:38.680603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.680492    4877 scope.go:122] "RemoveContainer" containerID="483a7d4e81846b6a74ea991ba3246266879a607a699318071cedfb2b4ec59919"
Oct 30 04:06:38.680603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.680385    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"
Oct 30 04:06:38.752756 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.752707    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr" podStartSLOduration=1.7526912860000001 podStartE2EDuration="1.752691286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:38.752121962 +0000 UTC m=+3827.181011404" watchObservedRunningTime="2026-10-30 04:06:38.752691286 +0000 UTC m=+3827.181580708"
Oct 30 04:06:38.800383 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.800355    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"]
Oct 30 04:06:38.812824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:38.812801    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6ad168-openshift-default-7987d79df9-7q7q4"]
Oct 30 04:06:39.692164 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:39.692125    4877 generic.go:356] "Generic (PLEG): container finished" podID="6a87329d-e60b-404a-95d2-b32e6fc6001a" containerID="ead0d30745c6cf89b51c64fe73a207a333e27a599b57601fe0ccf058c48fa7ca" exitCode=0
Oct 30 04:06:39.699756 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:39.699729    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" path="/var/lib/kubelet/pods/a07f1aad-625a-4cc5-b41c-6361b464df90/volumes"
Oct 30 04:06:39.700089 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:39.700072    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr" event={"ID":"6a87329d-e60b-404a-95d2-b32e6fc6001a","Type":"ContainerDied","Data":"ead0d30745c6cf89b51c64fe73a207a333e27a599b57601fe0ccf058c48fa7ca"}
Oct 30 04:06:40.316581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:40.316548    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"]
Oct 30 04:06:40.873963 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:40.873934    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:40.955496 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:40.949600    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x\" (UniqueName: \"kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x\") pod \"6a87329d-e60b-404a-95d2-b32e6fc6001a\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") "
Oct 30 04:06:40.955496 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:40.949861    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca\") pod \"6a87329d-e60b-404a-95d2-b32e6fc6001a\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") "
Oct 30 04:06:40.955496 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:40.949926    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca\") pod \"6a87329d-e60b-404a-95d2-b32e6fc6001a\" (UID: \"6a87329d-e60b-404a-95d2-b32e6fc6001a\") "
Oct 30 04:06:40.979448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:40.957690    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca" pod "6a87329d-e60b-404a-95d2-b32e6fc6001a" (UID: "6a87329d-e60b-404a-95d2-b32e6fc6001a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:41.029449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.020749    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x" pod "6a87329d-e60b-404a-95d2-b32e6fc6001a" (UID: "6a87329d-e60b-404a-95d2-b32e6fc6001a"). InnerVolumeSpecName "kube-api-access-tcv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:41.029449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.021242    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca" pod "6a87329d-e60b-404a-95d2-b32e6fc6001a" (UID: "6a87329d-e60b-404a-95d2-b32e6fc6001a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:41.058442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.051991    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:41.058442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.052027    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcv5x\" (UniqueName: \"kubernetes.io/projected/6a87329d-e60b-404a-95d2-b32e6fc6001a-kube-api-access-tcv5x\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:41.058442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.052041    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87329d-e60b-404a-95d2-b32e6fc6001a-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:41.715312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.715285    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"
Oct 30 04:06:41.715506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.715282    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr" event={"ID":"6a87329d-e60b-404a-95d2-b32e6fc6001a","Type":"ContainerDied","Data":"8d022c5c641c0fe2b519689738add86a8f8f1f965c6a642dc5115bc83add86c4"}
Oct 30 04:06:41.715506 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:41.715413    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d022c5c641c0fe2b519689738add86a8f8f1f965c6a642dc5115bc83add86c4"
Oct 30 04:06:43.113530 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.113453    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 04:06:43.218945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.218885    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.219587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.219556    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.219782 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.219468    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:43.219782 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.219771    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.219974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.219809    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.219974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.219863    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.219974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.219928    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.220155 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.220040    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.220155 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.220081    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.220846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.220808    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket\") pod \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\" (UID: \"ad6f100b-2891-446b-ab4d-5b7f535e9e00\") "
Oct 30 04:06:43.229600 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.228861    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:43.231044 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.231010    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.231177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.231056    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.234979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.234059    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "kube-api-access-4z2d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:43.234979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.234264    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:43.234979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.234451    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:43.234979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.234726    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:43.239013 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.238986    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:43.239447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.239392    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:06:43.243682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.243643    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token" pod "ad6f100b-2891-446b-ab4d-5b7f535e9e00" (UID: "ad6f100b-2891-446b-ab4d-5b7f535e9e00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:43.332839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332802    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.332839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332834    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.332839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332846    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.333126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332857    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.333126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332869    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2d5\" (UniqueName: \"kubernetes.io/projected/ad6f100b-2891-446b-ab4d-5b7f535e9e00-kube-api-access-4z2d5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.333126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332885    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.333126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.332896    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad6f100b-2891-446b-ab4d-5b7f535e9e00-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:43.736614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.736564    4877 generic.go:356] "Generic (PLEG): container finished" podID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerID="f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef" exitCode=0
Oct 30 04:06:43.736775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.736619    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"
Oct 30 04:06:43.736775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.736644    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" event={"ID":"ad6f100b-2891-446b-ab4d-5b7f535e9e00","Type":"ContainerDied","Data":"f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef"}
Oct 30 04:06:43.736775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.736699    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd" event={"ID":"ad6f100b-2891-446b-ab4d-5b7f535e9e00","Type":"ContainerDied","Data":"d5f5d41066c73e4452240c478797b5e65d465b141b93cf5d6def6954b5538445"}
Oct 30 04:06:43.736775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.736726    4877 scope.go:122] "RemoveContainer" containerID="f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef"
Oct 30 04:06:43.748965 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.748945    4877 scope.go:122] "RemoveContainer" containerID="d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"
Oct 30 04:06:43.759414 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.759391    4877 scope.go:122] "RemoveContainer" containerID="f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef"
Oct 30 04:06:43.759714 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:43.759694    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef\": container with ID starting with f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef not found: ID does not exist" containerID="f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef"
Oct 30 04:06:43.759770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.759723    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef"} err="failed to get container status \"f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef\": rpc error: code = NotFound desc = could not find container \"f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef\": container with ID starting with f7e7f0a4f5092772949e390adcca04ed7c7b124a872e93fc4f9d9b64e69ddbef not found: ID does not exist"
Oct 30 04:06:43.759770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.759747    4877 scope.go:122] "RemoveContainer" containerID="d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"
Oct 30 04:06:43.760052 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:43.760028    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a\": container with ID starting with d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a not found: ID does not exist" containerID="d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"
Oct 30 04:06:43.760117 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.760057    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a"} err="failed to get container status \"d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a\": rpc error: code = NotFound desc = could not find container \"d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a\": container with ID starting with d451360afba772f323b85e07b069202ae72fca532f2017ce837ba973df364f8a not found: ID does not exist"
Oct 30 04:06:43.795291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.795262    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"]
Oct 30 04:06:43.805370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:43.805334    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-gh4qd"]
Oct 30 04:06:44.472091 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:44.472054    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"]
Oct 30 04:06:45.317537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:45.317496    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" containerName="payload-pre-processing" containerID="cri-o://2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15" gracePeriod=25
Oct 30 04:06:45.700816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:45.700362    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" path="/var/lib/kubelet/pods/ad6f100b-2891-446b-ab4d-5b7f535e9e00/volumes"
Oct 30 04:06:46.237320 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:46.237286    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:06:46.257125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.257103    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 04:06:46.295349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.295208    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume\") pod \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") "
Oct 30 04:06:46.296812 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.295826    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245\" (UniqueName: \"kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245\") pod \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\" (UID: \"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a\") "
Oct 30 04:06:46.297378 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.297336    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume" pod "0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" (UID: "0baa73a5-9e5f-4aa3-aaea-97c02c952e3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:46.311481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.311400    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245" pod "0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" (UID: "0baa73a5-9e5f-4aa3-aaea-97c02c952e3a"). InnerVolumeSpecName "kube-api-access-ss245". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:46.397455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.397413    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss245\" (UniqueName: \"kubernetes.io/projected/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-kube-api-access-ss245\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:46.397455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.397451    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:46.763675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.763641    4877 generic.go:356] "Generic (PLEG): container finished" podID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerID="e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62" exitCode=0
Oct 30 04:06:46.764145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.763692    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"
Oct 30 04:06:46.764145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.763703    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" event={"ID":"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a","Type":"ContainerDied","Data":"e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62"}
Oct 30 04:06:46.764145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.763743    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn" event={"ID":"0baa73a5-9e5f-4aa3-aaea-97c02c952e3a","Type":"ContainerDied","Data":"fe0cd9da5ad3789b094ae09d8eb04be39b47a05d1f15781c9261135ca2b53e5a"}
Oct 30 04:06:46.764145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.763766    4877 scope.go:122] "RemoveContainer" containerID="e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62"
Oct 30 04:06:46.775004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.774983    4877 scope.go:122] "RemoveContainer" containerID="e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62"
Oct 30 04:06:46.775323 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:46.775301    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62\": container with ID starting with e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62 not found: ID does not exist" containerID="e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62"
Oct 30 04:06:46.775449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.775328    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62"} err="failed to get container status \"e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62\": rpc error: code = NotFound desc = could not find container \"e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62\": container with ID starting with e298326cc7934817ba60421bb8b6c8dbbfaa72ee095ed1640a03cba7c97c8f62 not found: ID does not exist"
Oct 30 04:06:46.812116 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.812086    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"]
Oct 30 04:06:46.824052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:46.824023    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-1a1b45-84568cdd5f-bq6zn"]
Oct 30 04:06:47.699321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:47.699281    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" path="/var/lib/kubelet/pods/0baa73a5-9e5f-4aa3-aaea-97c02c952e3a/volumes"
Oct 30 04:06:49.472474 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:49.472386    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerName="payload-pre-processing" containerID="cri-o://504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151" gracePeriod=25
Oct 30 04:06:51.065062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:51.065013    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:06:53.522670 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:53.522634    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"]
Oct 30 04:06:53.523744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:53.523707    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" containerID="cri-o://3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5" gracePeriod=30
Oct 30 04:06:54.451672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.451636    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"]
Oct 30 04:06:54.452523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452490    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy"
Oct 30 04:06:54.452523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452511    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452530    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452537    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452558    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452565    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452601    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerName="payload-pre-processing"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452611    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerName="payload-pre-processing"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452624    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6a87329d-e60b-404a-95d2-b32e6fc6001a" containerName="revoke-keys"
Oct 30 04:06:54.452731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452631    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a87329d-e60b-404a-95d2-b32e6fc6001a" containerName="revoke-keys"
Oct 30 04:06:54.453177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452797    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy"
Oct 30 04:06:54.453177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452811    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy"
Oct 30 04:06:54.453177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452827    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baa73a5-9e5f-4aa3-aaea-97c02c952e3a" containerName="payload-pre-processing"
Oct 30 04:06:54.453177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.452837    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a87329d-e60b-404a-95d2-b32e6fc6001a" containerName="revoke-keys"
Oct 30 04:06:54.458247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.458226    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.470668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.470640    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"]
Oct 30 04:06:54.475818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.475797    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:06:54.476006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.475987    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:06:54.521862 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.521829    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.522060 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.521912    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.522135 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.522112    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk54z\" (UniqueName: \"kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.623362 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.623316    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk54z\" (UniqueName: \"kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.623854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.623417    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.623854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.623505    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.624260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.624233    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.625874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.625845    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.641453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.641414    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk54z\" (UniqueName: \"kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z\") pod \"maas-api-revoke-sub-e3f7cacc-4wstn\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:54.773773 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:54.773747    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:55.138690 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:55.138657    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"]
Oct 30 04:06:56.158503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:56.157805    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn" event={"ID":"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab","Type":"ContainerStarted","Data":"720d11482213a245e883a0942202b4ed1be498d9d54b6f87569ac0411ddd55e6"}
Oct 30 04:06:56.158503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:56.157876    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn" event={"ID":"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab","Type":"ContainerStarted","Data":"304ba4a74d204d8295e90fff3d3f91a7055a67796f484bfd6861813ebf8ea514"}
Oct 30 04:06:56.244043 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:56.243984    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn" podStartSLOduration=2.243964006 podStartE2EDuration="2.243964006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:56.240938052 +0000 UTC m=+3844.669827469" watchObservedRunningTime="2026-10-30 04:06:56.243964006 +0000 UTC m=+3844.672853431"
Oct 30 04:06:57.167699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:57.167666    4877 generic.go:356] "Generic (PLEG): container finished" podID="713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" containerID="720d11482213a245e883a0942202b4ed1be498d9d54b6f87569ac0411ddd55e6" exitCode=0
Oct 30 04:06:57.168115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:57.167747    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn" event={"ID":"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab","Type":"ContainerDied","Data":"720d11482213a245e883a0942202b4ed1be498d9d54b6f87569ac0411ddd55e6"}
Oct 30 04:06:57.219510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:57.219471    4877 scope.go:122] "RemoveContainer" containerID="957592513e890b53437f2f84e7994c3b556b46bcaf756001b3c8ae38b0128c6f"
Oct 30 04:06:58.333520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.333479    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:06:58.333894 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.333556    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:06:58.380814 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.380775    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:58.480271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.480169    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca\") pod \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") "
Oct 30 04:06:58.480467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.480356    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca\") pod \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") "
Oct 30 04:06:58.480552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.480498    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z\" (UniqueName: \"kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z\") pod \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\" (UID: \"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab\") "
Oct 30 04:06:58.481861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.481828    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca" pod "713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" (UID: "713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:58.483675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.483641    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z" pod "713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" (UID: "713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab"). InnerVolumeSpecName "kube-api-access-hk54z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:58.485933 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.485902    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca" pod "713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" (UID: "713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:58.582738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.582702    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:58.582738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.582739    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk54z\" (UniqueName: \"kubernetes.io/projected/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-kube-api-access-hk54z\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:58.582974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.582757    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:58.655340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.655304    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:06:58.856399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.856088    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 04:06:58.978188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.978148    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"]
Oct 30 04:06:58.986614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.986581    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"]
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989176    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989266    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989321    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989364    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989409    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989511    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:58.989659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.989618    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.991392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.991019    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:58.991392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.991260    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:58.991656 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.991622    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.991797 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.991730    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.991797 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.991763    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert\") pod \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\" (UID: \"19a12af6-dfbf-4ee3-b76e-610e2c76f75f\") "
Oct 30 04:06:58.992354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.992333    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:58.992481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.992357    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:58.992481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.992371    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:58.992481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.992441    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:58.992729 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.992708    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:58.993162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.993117    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:06:58.993485 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.993464    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "kube-api-access-sgs7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:58.997508 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.994999    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:58.998846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:58.998819    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy" pod "19a12af6-dfbf-4ee3-b76e-610e2c76f75f" (UID: "19a12af6-dfbf-4ee3-b76e-610e2c76f75f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:59.097343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.097236    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgs7v\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-kube-api-access-sgs7v\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:59.097343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.097288    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:59.097343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.097306    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:59.097343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.097334    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:59.097757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.097349    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:59.097757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.097363    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a12af6-dfbf-4ee3-b76e-610e2c76f75f-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:06:59.187088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.187047    4877 generic.go:356] "Generic (PLEG): container finished" podID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerID="3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5" exitCode=0
Oct 30 04:06:59.187295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.187082    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" event={"ID":"19a12af6-dfbf-4ee3-b76e-610e2c76f75f","Type":"ContainerDied","Data":"3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5"}
Oct 30 04:06:59.187295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.187111    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"
Oct 30 04:06:59.187295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.187153    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f" event={"ID":"19a12af6-dfbf-4ee3-b76e-610e2c76f75f","Type":"ContainerDied","Data":"a6f0cc7d9063ae1fa6803e446e4b5e3d35ecd3eb1310fb0786fbb3e05c515352"}
Oct 30 04:06:59.187295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.187184    4877 scope.go:122] "RemoveContainer" containerID="3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5"
Oct 30 04:06:59.191157 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.190687    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn" event={"ID":"713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab","Type":"ContainerDied","Data":"304ba4a74d204d8295e90fff3d3f91a7055a67796f484bfd6861813ebf8ea514"}
Oct 30 04:06:59.191157 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.190738    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="304ba4a74d204d8295e90fff3d3f91a7055a67796f484bfd6861813ebf8ea514"
Oct 30 04:06:59.191157 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.190801    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"
Oct 30 04:06:59.214487 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.213326    4877 scope.go:122] "RemoveContainer" containerID="f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"
Oct 30 04:06:59.246092 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.246071    4877 scope.go:122] "RemoveContainer" containerID="3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5"
Oct 30 04:06:59.246417 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:59.246393    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5\": container with ID starting with 3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5 not found: ID does not exist" containerID="3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5"
Oct 30 04:06:59.246417 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.246440    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5"} err="failed to get container status \"3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5\": rpc error: code = NotFound desc = could not find container \"3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5\": container with ID starting with 3b9718c2a87c6c072781ddd9cf446ab2d84755e08088a4c16c9f2812f3a458b5 not found: ID does not exist"
Oct 30 04:06:59.246660 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.246466    4877 scope.go:122] "RemoveContainer" containerID="f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"
Oct 30 04:06:59.246742 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:06:59.246717    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259\": container with ID starting with f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259 not found: ID does not exist" containerID="f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"
Oct 30 04:06:59.246810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.246751    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259"} err="failed to get container status \"f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259\": rpc error: code = NotFound desc = could not find container \"f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259\": container with ID starting with f8186b6a6b3df7ee20278f0bcfa308994325f738ae93b0dd20c511a528920259 not found: ID does not exist"
Oct 30 04:06:59.264069 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.264043    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"]
Oct 30 04:06:59.272009 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.271981    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-vkh9f"]
Oct 30 04:06:59.703765 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:06:59.703687    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" path="/var/lib/kubelet/pods/19a12af6-dfbf-4ee3-b76e-610e2c76f75f/volumes"
Oct 30 04:07:00.616822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:00.616782    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"]
Oct 30 04:07:00.617890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:00.617855    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy" containerID="cri-o://9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f" gracePeriod=30
Oct 30 04:07:01.060472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:01.060430    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:02.222794 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:02.222760    4877 generic.go:356] "Generic (PLEG): container finished" podID="f435747d-c138-430d-9a05-c3789c910abf" containerID="963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da" exitCode=0
Oct 30 04:07:02.223231 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:02.222837    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" event={"ID":"f435747d-c138-430d-9a05-c3789c910abf","Type":"ContainerDied","Data":"963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"}
Oct 30 04:07:02.223303 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:02.223288    4877 scope.go:122] "RemoveContainer" containerID="963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"
Oct 30 04:07:02.225007 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:02.224979    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:07:02.225085 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:02.225047    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:07:02.225085 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:02.225078    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:07:03.235644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:03.235608    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" event={"ID":"f435747d-c138-430d-9a05-c3789c910abf","Type":"ContainerStarted","Data":"4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c"}
Oct 30 04:07:03.326401 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:03.326364    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:07:03.326620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:03.326592    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 04:07:03.326675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:03.326649    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 04:07:04.326619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:04.326582    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 04:07:04.327027 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:04.326647    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 04:07:05.326497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.326467    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 04:07:05.326631 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.326547    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 04:07:05.741837 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.741810    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 04:07:05.843387 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.843349    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms\" (UniqueName: \"kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms\") pod \"00d38f70-a370-4375-8264-8e4e55a446bf\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") "
Oct 30 04:07:05.843796 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.843773    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume\") pod \"00d38f70-a370-4375-8264-8e4e55a446bf\" (UID: \"00d38f70-a370-4375-8264-8e4e55a446bf\") "
Oct 30 04:07:05.844168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.844137    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume" pod "00d38f70-a370-4375-8264-8e4e55a446bf" (UID: "00d38f70-a370-4375-8264-8e4e55a446bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:05.844628 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.844577    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d38f70-a370-4375-8264-8e4e55a446bf-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:05.863444 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.852827    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms" pod "00d38f70-a370-4375-8264-8e4e55a446bf" (UID: "00d38f70-a370-4375-8264-8e4e55a446bf"). InnerVolumeSpecName "kube-api-access-m5dms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:05.946406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.946379    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5dms\" (UniqueName: \"kubernetes.io/projected/00d38f70-a370-4375-8264-8e4e55a446bf-kube-api-access-m5dms\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:05.947227 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:05.947207    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047028    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047117    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047189    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047274    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047312    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047361    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047401    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.047835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047758    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:06.048471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.047897    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:06.048471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.048000    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:06.048471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.048146    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:06.048471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.048205    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.048866    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.048917    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk\") pod \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\" (UID: \"b8566642-4ab2-4ded-ae1a-75bc8d2fb796\") "
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.049504    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.049526    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.049541    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.049554    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.049592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.049568    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.050188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.050161    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:07:06.050942 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.050909    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:06.051575 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.051549    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:06.052847 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.052810    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk" pod "b8566642-4ab2-4ded-ae1a-75bc8d2fb796" (UID: "b8566642-4ab2-4ded-ae1a-75bc8d2fb796"). InnerVolumeSpecName "kube-api-access-fd8vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:06.150603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.150569    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.150603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.150598    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd8vk\" (UniqueName: \"kubernetes.io/projected/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-kube-api-access-fd8vk\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.150603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.150610    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.150859 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.150619    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8566642-4ab2-4ded-ae1a-75bc8d2fb796-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:06.212638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.212545    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"]
Oct 30 04:07:06.213300 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213276    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy"
Oct 30 04:07:06.213300 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213301    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213318    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213325    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213365    4877 cpu_manager.go:414] "containerMap: removing container" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213373    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213392    4877 cpu_manager.go:414] "containerMap: removing container" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213398    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213440    4877 cpu_manager.go:414] "containerMap: removing container" podUID="713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" containerName="revoke-keys"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213448    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" containerName="revoke-keys"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213460    4877 cpu_manager.go:414] "containerMap: removing container" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" containerName="payload-pre-processing"
Oct 30 04:07:06.213523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213468    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" containerName="payload-pre-processing"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213648    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213666    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" containerName="revoke-keys"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213681    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" containerName="payload-pre-processing"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213711    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213722    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213735    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6f100b-2891-446b-ab4d-5b7f535e9e00" containerName="istio-proxy"
Oct 30 04:07:06.214040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.213743    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07f1aad-625a-4cc5-b41c-6361b464df90" containerName="istio-proxy"
Oct 30 04:07:06.218587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.218219    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.222914 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.222880    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"]
Oct 30 04:07:06.232141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.232066    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:07:06.232141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.232106    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:07:06.263095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.263067    4877 generic.go:356] "Generic (PLEG): container finished" podID="00d38f70-a370-4375-8264-8e4e55a446bf" containerID="2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15" exitCode=0
Oct 30 04:07:06.263240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.263184    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" event={"ID":"00d38f70-a370-4375-8264-8e4e55a446bf","Type":"ContainerDied","Data":"2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15"}
Oct 30 04:07:06.263240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.263221    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x" event={"ID":"00d38f70-a370-4375-8264-8e4e55a446bf","Type":"ContainerDied","Data":"51755c4948f800dcbd96f72e7ca9045b597a67d1bb808432cd3c71ee10b37a09"}
Oct 30 04:07:06.263320 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.263241    4877 scope.go:122] "RemoveContainer" containerID="2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15"
Oct 30 04:07:06.263357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.263332    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"
Oct 30 04:07:06.265637 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.265617    4877 generic.go:356] "Generic (PLEG): container finished" podID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerID="9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f" exitCode=0
Oct 30 04:07:06.265889 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.265875    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" event={"ID":"b8566642-4ab2-4ded-ae1a-75bc8d2fb796","Type":"ContainerDied","Data":"9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f"}
Oct 30 04:07:06.265999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.265986    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t" event={"ID":"b8566642-4ab2-4ded-ae1a-75bc8d2fb796","Type":"ContainerDied","Data":"cdd8fe4c2a01d0655b3d1eaf7d23cb0800674e786a9bfbbc2a5cae4d432702c6"}
Oct 30 04:07:06.266123 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.266111    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"
Oct 30 04:07:06.322394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.322362    4877 scope.go:122] "RemoveContainer" containerID="2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15"
Oct 30 04:07:06.323686 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:06.323632    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15\": container with ID starting with 2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15 not found: ID does not exist" containerID="2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15"
Oct 30 04:07:06.330333 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.330298    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 04:07:06.330473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.330377    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 04:07:06.330544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.323701    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15"} err="failed to get container status \"2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15\": rpc error: code = NotFound desc = could not find container \"2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15\": container with ID starting with 2ed3eaf31794a972fa7fe9bd93c89524dfd5363df3186ac1ccd9bae69537eb15 not found: ID does not exist"
Oct 30 04:07:06.330544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.330531    4877 scope.go:122] "RemoveContainer" containerID="9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f"
Oct 30 04:07:06.355055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.355019    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.355219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.355149    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.355219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.355201    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdm5\" (UniqueName: \"kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.381635 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.376256    4877 scope.go:122] "RemoveContainer" containerID="703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"
Oct 30 04:07:06.428115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.427904    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"]
Oct 30 04:07:06.438525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.438414    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-35f43f-openshift-default-69bf4bcd4f-lh44t"]
Oct 30 04:07:06.441613 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.441593    4877 scope.go:122] "RemoveContainer" containerID="9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f"
Oct 30 04:07:06.442456 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:06.442404    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f\": container with ID starting with 9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f not found: ID does not exist" containerID="9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f"
Oct 30 04:07:06.442655 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.442623    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f"} err="failed to get container status \"9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f\": rpc error: code = NotFound desc = could not find container \"9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f\": container with ID starting with 9b945c92a08cf19f8036fabb0ada9566392b399d22ce72565d7540e379c7a74f not found: ID does not exist"
Oct 30 04:07:06.442785 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.442773    4877 scope.go:122] "RemoveContainer" containerID="703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"
Oct 30 04:07:06.443368 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.443346    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"]
Oct 30 04:07:06.443800 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:06.443770    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286\": container with ID starting with 703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286 not found: ID does not exist" containerID="703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"
Oct 30 04:07:06.443915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.443808    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286"} err="failed to get container status \"703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286\": rpc error: code = NotFound desc = could not find container \"703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286\": container with ID starting with 703296e2239e401623613bd7ffcdc3085ae9e935da15b9652dc601d0f4965286 not found: ID does not exist"
Oct 30 04:07:06.449461 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.449029    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-817531-7bdf4dc867-r9h7x"]
Oct 30 04:07:06.456532 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.456509    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.456681 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.456665    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdm5\" (UniqueName: \"kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.456889 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.456876    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.457906 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.457887    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.459673 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.459650    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.474916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.474864    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdm5\" (UniqueName: \"kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5\") pod \"maas-api-revoke-sub-a72ea379-v8tc6\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:06.537289 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:06.537256    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:07.073336 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:07:07.073312    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a46e90_0bd9_4f85_ada5_cae7c03b074b.slice/crio-12757b5b256ad8f46215b3f0eb69e1e1dfccaa63d2eb71ba3e34db47ce838169 WatchSource:0}: Error finding container 12757b5b256ad8f46215b3f0eb69e1e1dfccaa63d2eb71ba3e34db47ce838169: Status 404 returned error can't find the container with id 12757b5b256ad8f46215b3f0eb69e1e1dfccaa63d2eb71ba3e34db47ce838169
Oct 30 04:07:07.075208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:07.075138    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"]
Oct 30 04:07:07.330125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:07.328016    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 04:07:07.330125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:07.328087    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 04:07:07.341280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:07.340934    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6" event={"ID":"65a46e90-0bd9-4f85-ada5-cae7c03b074b","Type":"ContainerStarted","Data":"12757b5b256ad8f46215b3f0eb69e1e1dfccaa63d2eb71ba3e34db47ce838169"}
Oct 30 04:07:07.708126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:07.708042    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d38f70-a370-4375-8264-8e4e55a446bf" path="/var/lib/kubelet/pods/00d38f70-a370-4375-8264-8e4e55a446bf/volumes"
Oct 30 04:07:07.708574 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:07.708559    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" path="/var/lib/kubelet/pods/b8566642-4ab2-4ded-ae1a-75bc8d2fb796/volumes"
Oct 30 04:07:08.325883 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.325855    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Oct 30 04:07:08.326290 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.325911    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Oct 30 04:07:08.351549 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.351510    4877 generic.go:356] "Generic (PLEG): container finished" podID="65a46e90-0bd9-4f85-ada5-cae7c03b074b" containerID="b2a86cae0ce5eaee96b32534e075685b287ec4d4f3e9c502715ad5e1d9028d38" exitCode=0
Oct 30 04:07:08.351731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.351599    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6" event={"ID":"65a46e90-0bd9-4f85-ada5-cae7c03b074b","Type":"ContainerDied","Data":"b2a86cae0ce5eaee96b32534e075685b287ec4d4f3e9c502715ad5e1d9028d38"}
Oct 30 04:07:08.467525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.466846    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"]
Oct 30 04:07:08.467525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.467081    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" containerID="cri-o://58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693" gracePeriod=30
Oct 30 04:07:08.654947 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.654861    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:08.955400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:08.946472    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"]
Oct 30 04:07:09.033410 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.012643    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"]
Oct 30 04:07:09.297751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.297721    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:07:09.330939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.330911    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:07:09.362690 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.362667    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:07:09.693532 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.693501    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:09.822269 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.822247    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 04:07:09.881894 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.881846    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca\") pod \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") "
Oct 30 04:07:09.882080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.881962    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca\") pod \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") "
Oct 30 04:07:09.882080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.882005    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5\" (UniqueName: \"kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5\") pod \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\" (UID: \"65a46e90-0bd9-4f85-ada5-cae7c03b074b\") "
Oct 30 04:07:09.882377 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.882344    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca" pod "65a46e90-0bd9-4f85-ada5-cae7c03b074b" (UID: "65a46e90-0bd9-4f85-ada5-cae7c03b074b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:09.882907 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.882879    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:09.884485 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.884391    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5" pod "65a46e90-0bd9-4f85-ada5-cae7c03b074b" (UID: "65a46e90-0bd9-4f85-ada5-cae7c03b074b"). InnerVolumeSpecName "kube-api-access-hzdm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:09.884698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.884678    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca" pod "65a46e90-0bd9-4f85-ada5-cae7c03b074b" (UID: "65a46e90-0bd9-4f85-ada5-cae7c03b074b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:09.983763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.983724    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume\") pod \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") "
Oct 30 04:07:09.983947 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.983815    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t\" (UniqueName: \"kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t\") pod \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\" (UID: \"6d74e3bf-7840-4a6c-a58b-1941d7a29da0\") "
Oct 30 04:07:09.984131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.984099    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume" pod "6d74e3bf-7840-4a6c-a58b-1941d7a29da0" (UID: "6d74e3bf-7840-4a6c-a58b-1941d7a29da0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:09.984334 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.984317    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdm5\" (UniqueName: \"kubernetes.io/projected/65a46e90-0bd9-4f85-ada5-cae7c03b074b-kube-api-access-hzdm5\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:09.984400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.984336    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:09.984400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.984346    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a46e90-0bd9-4f85-ada5-cae7c03b074b-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:09.985876 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:09.985854    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t" pod "6d74e3bf-7840-4a6c-a58b-1941d7a29da0" (UID: "6d74e3bf-7840-4a6c-a58b-1941d7a29da0"). InnerVolumeSpecName "kube-api-access-dxg6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:10.085956 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.085873    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxg6t\" (UniqueName: \"kubernetes.io/projected/6d74e3bf-7840-4a6c-a58b-1941d7a29da0-kube-api-access-dxg6t\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:10.371804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.371715    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"
Oct 30 04:07:10.372220 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.371709    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6" event={"ID":"65a46e90-0bd9-4f85-ada5-cae7c03b074b","Type":"ContainerDied","Data":"12757b5b256ad8f46215b3f0eb69e1e1dfccaa63d2eb71ba3e34db47ce838169"}
Oct 30 04:07:10.372220 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.371878    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12757b5b256ad8f46215b3f0eb69e1e1dfccaa63d2eb71ba3e34db47ce838169"
Oct 30 04:07:10.373573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.373546    4877 generic.go:356] "Generic (PLEG): container finished" podID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerID="504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151" exitCode=0
Oct 30 04:07:10.373732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.373601    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"
Oct 30 04:07:10.373732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.373616    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" event={"ID":"6d74e3bf-7840-4a6c-a58b-1941d7a29da0","Type":"ContainerDied","Data":"504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151"}
Oct 30 04:07:10.373732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.373717    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4" event={"ID":"6d74e3bf-7840-4a6c-a58b-1941d7a29da0","Type":"ContainerDied","Data":"2c973c5291e24141a9bfef8bc8ec2a75a074b5b6043ab9e29ed6d9796386cba7"}
Oct 30 04:07:10.373908 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.373738    4877 scope.go:122] "RemoveContainer" containerID="504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151"
Oct 30 04:07:10.387671 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.387652    4877 scope.go:122] "RemoveContainer" containerID="504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151"
Oct 30 04:07:10.387917 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:10.387897    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151\": container with ID starting with 504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151 not found: ID does not exist" containerID="504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151"
Oct 30 04:07:10.387988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.387928    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151"} err="failed to get container status \"504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151\": rpc error: code = NotFound desc = could not find container \"504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151\": container with ID starting with 504b0595ea75d56853b0d5c04b723dc0e8b3dab2031a8500fb54ba89551ff151 not found: ID does not exist"
Oct 30 04:07:10.419616 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.419592    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"]
Oct 30 04:07:10.429213 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:10.429191    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-547a78-5d6f566cf5-x5nl4"]
Oct 30 04:07:11.703145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:11.703108    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" path="/var/lib/kubelet/pods/6d74e3bf-7840-4a6c-a58b-1941d7a29da0/volumes"
Oct 30 04:07:12.028320 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:12.028287    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"]
Oct 30 04:07:12.037406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:12.037381    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc16f1-8brpz"]
Oct 30 04:07:12.477744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:12.477655    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:07:12.477744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:12.477719    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:07:13.313445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.298139    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"]
Oct 30 04:07:13.313445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.298366    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" containerID="cri-o://53aecb49627b3294e058397fdbe7291cf11faa26dd47688d7cbb20313a34ba22" gracePeriod=30
Oct 30 04:07:13.710312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.710275    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8837a87-50fc-4fc8-b285-dc34a615dd52" path="/var/lib/kubelet/pods/f8837a87-50fc-4fc8-b285-dc34a615dd52/volumes"
Oct 30 04:07:13.775351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.775323    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 04:07:13.847089 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847005    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847089 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847071    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847116    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847205    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847275    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847323    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847354    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847363    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:13.847606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847383    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847513    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket\") pod \"bf4348e6-070a-4199-952b-58649a0ef0f6\" (UID: \"bf4348e6-070a-4199-952b-58649a0ef0f6\") "
Oct 30 04:07:13.847606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847592    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:13.847879 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847757    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:13.847989 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.847965    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:13.848238 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.848215    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.848331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.848246    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.848331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.848262    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.848331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.848277    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.848331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.848288    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:13.849903 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.849874    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:13.850008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.849974    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "kube-api-access-dvcb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:13.850290 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.850269    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:13.851002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.850977    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo" pod "bf4348e6-070a-4199-952b-58649a0ef0f6" (UID: "bf4348e6-070a-4199-952b-58649a0ef0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:07:13.946789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.946755    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85" podUID="4f31e566-4b61-40b3-a685-4dd0c4bfce2b" containerName="payload-processing" containerID="cri-o://c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742" gracePeriod=25
Oct 30 04:07:13.949953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.949927    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvcb8\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-kube-api-access-dvcb8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.949953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.949956    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.950082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.949966    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.950082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.949975    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4348e6-070a-4199-952b-58649a0ef0f6-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:13.950082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:13.949983    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4348e6-070a-4199-952b-58649a0ef0f6-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.036771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.035830    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2" podUID="1ff8c4ad-1da5-44ac-9f36-75dce02091c4" containerName="payload-pre-processing" containerID="cri-o://b9aac0be858f2db1b1170ebbdac052fe19a1bb435b65f6f5693a91e3ff5a9cdd" gracePeriod=25
Oct 30 04:07:14.062903 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.062864    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main" containerID="cri-o://d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c" gracePeriod=45
Oct 30 04:07:14.063047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.062911    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" containerID="cri-o://e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff" gracePeriod=45
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.232330    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"]
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233347    4877 cpu_manager.go:414] "containerMap: removing container" podUID="65a46e90-0bd9-4f85-ada5-cae7c03b074b" containerName="revoke-keys"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233363    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a46e90-0bd9-4f85-ada5-cae7c03b074b" containerName="revoke-keys"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233390    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233396    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233413    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerName="payload-pre-processing"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233418    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerName="payload-pre-processing"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233445    4877 cpu_manager.go:414] "containerMap: removing container" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233452    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8566642-4ab2-4ded-ae1a-75bc8d2fb796" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233466    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233471    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233567    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a12af6-dfbf-4ee3-b76e-610e2c76f75f" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233578    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a46e90-0bd9-4f85-ada5-cae7c03b074b" containerName="revoke-keys"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233592    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233601    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d74e3bf-7840-4a6c-a58b-1941d7a29da0" containerName="payload-pre-processing"
Oct 30 04:07:14.238619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.233609    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerName="istio-proxy"
Oct 30 04:07:14.239591 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.238769    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.260138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.256151    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"]
Oct 30 04:07:14.260138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.257246    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.260138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.257355    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkcp\" (UniqueName: \"kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.260138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.257471    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.281066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.281025    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": dial tcp 10.133.0.81:8000: connect: connection refused"
Oct 30 04:07:14.308983 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.308933    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:07:14.309484 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.309160    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:07:14.358912 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.358834    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.358989    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.359074    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkcp\" (UniqueName: \"kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.362594 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.362558    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.395319 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.372289    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.395527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.378654    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 04:07:14.395682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.384787    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkcp\" (UniqueName: \"kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp\") pod \"maas-api-revoke-sub-151512ce-p9fc8\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.422726 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.422658    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 04:07:14.459793 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.459761    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache\") pod \"3ee1492c-d2c2-4e86-818b-514ac3368623\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") "
Oct 30 04:07:14.459965 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.459800    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home\") pod \"3ee1492c-d2c2-4e86-818b-514ac3368623\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") "
Oct 30 04:07:14.459965 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.459826    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn\" (UniqueName: \"kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn\") pod \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") "
Oct 30 04:07:14.459965 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.459864    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs\") pod \"3ee1492c-d2c2-4e86-818b-514ac3368623\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") "
Oct 30 04:07:14.460172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.460074    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home" pod "3ee1492c-d2c2-4e86-818b-514ac3368623" (UID: "3ee1492c-d2c2-4e86-818b-514ac3368623"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.460172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.460147    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache" pod "3ee1492c-d2c2-4e86-818b-514ac3368623" (UID: "3ee1492c-d2c2-4e86-818b-514ac3368623"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.460284 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.460204    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n\" (UniqueName: \"kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n\") pod \"3ee1492c-d2c2-4e86-818b-514ac3368623\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") "
Oct 30 04:07:14.460284 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.460255    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm\") pod \"3ee1492c-d2c2-4e86-818b-514ac3368623\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") "
Oct 30 04:07:14.461545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.460462    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume\") pod \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\" (UID: \"4f31e566-4b61-40b3-a685-4dd0c4bfce2b\") "
Oct 30 04:07:14.461545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.460515    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir\") pod \"3ee1492c-d2c2-4e86-818b-514ac3368623\" (UID: \"3ee1492c-d2c2-4e86-818b-514ac3368623\") "
Oct 30 04:07:14.461545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.461308    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.461545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.461331    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.461545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.461540    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir" pod "3ee1492c-d2c2-4e86-818b-514ac3368623" (UID: "3ee1492c-d2c2-4e86-818b-514ac3368623"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.463272 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.463051    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n" pod "3ee1492c-d2c2-4e86-818b-514ac3368623" (UID: "3ee1492c-d2c2-4e86-818b-514ac3368623"). InnerVolumeSpecName "kube-api-access-p5k6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:14.463348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.463301    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume" pod "4f31e566-4b61-40b3-a685-4dd0c4bfce2b" (UID: "4f31e566-4b61-40b3-a685-4dd0c4bfce2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:14.486370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.486335    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn" pod "4f31e566-4b61-40b3-a685-4dd0c4bfce2b" (UID: "4f31e566-4b61-40b3-a685-4dd0c4bfce2b"). InnerVolumeSpecName "kube-api-access-l75sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:14.490478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.490416    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm" pod "3ee1492c-d2c2-4e86-818b-514ac3368623" (UID: "3ee1492c-d2c2-4e86-818b-514ac3368623"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.496331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.496211    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs" pod "3ee1492c-d2c2-4e86-818b-514ac3368623" (UID: "3ee1492c-d2c2-4e86-818b-514ac3368623"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:14.504050 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.503998    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 04:07:14.506201 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.506175    4877 generic.go:356] "Generic (PLEG): container finished" podID="bf4348e6-070a-4199-952b-58649a0ef0f6" containerID="58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693" exitCode=0
Oct 30 04:07:14.506471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.506448    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" event={"ID":"bf4348e6-070a-4199-952b-58649a0ef0f6","Type":"ContainerDied","Data":"58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693"}
Oct 30 04:07:14.507311 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.506691    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"
Oct 30 04:07:14.507769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.507746    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd" event={"ID":"bf4348e6-070a-4199-952b-58649a0ef0f6","Type":"ContainerDied","Data":"fb1d836e3e8b059329c191a295c7b0c8b7364f6e81bfe140f949d9b19c5eb4fc"}
Oct 30 04:07:14.507960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.507905    4877 scope.go:122] "RemoveContainer" containerID="58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693"
Oct 30 04:07:14.508436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.508252    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 04:07:14.516843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.516823    4877 generic.go:356] "Generic (PLEG): container finished" podID="4f31e566-4b61-40b3-a685-4dd0c4bfce2b" containerID="c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742" exitCode=0
Oct 30 04:07:14.517210 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.516979    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85" event={"ID":"4f31e566-4b61-40b3-a685-4dd0c4bfce2b","Type":"ContainerDied","Data":"c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742"}
Oct 30 04:07:14.518934 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.518920    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85" event={"ID":"4f31e566-4b61-40b3-a685-4dd0c4bfce2b","Type":"ContainerDied","Data":"3ae0470d7c32fab0a673ad3bc48929d6ee5798366ff52f81ff79c52faa7a9d01"}
Oct 30 04:07:14.519328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.519313    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" event={"ID":"3ee1492c-d2c2-4e86-818b-514ac3368623","Type":"ContainerDied","Data":"d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c"}
Oct 30 04:07:14.519414 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.518820    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"
Oct 30 04:07:14.521345 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.517045    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"
Oct 30 04:07:14.522569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.518770    4877 generic.go:356] "Generic (PLEG): container finished" podID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerID="d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c" exitCode=0
Oct 30 04:07:14.522661 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.522644    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv" event={"ID":"3ee1492c-d2c2-4e86-818b-514ac3368623","Type":"ContainerDied","Data":"0cc684a72bd9c4bbd318366b78bc520fcb3b86235664259a27752bbb731324a1"}
Oct 30 04:07:14.532383 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.530352    4877 generic.go:356] "Generic (PLEG): container finished" podID="1ff8c4ad-1da5-44ac-9f36-75dce02091c4" containerID="b9aac0be858f2db1b1170ebbdac052fe19a1bb435b65f6f5693a91e3ff5a9cdd" exitCode=0
Oct 30 04:07:14.532383 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.530494    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2" event={"ID":"1ff8c4ad-1da5-44ac-9f36-75dce02091c4","Type":"ContainerDied","Data":"b9aac0be858f2db1b1170ebbdac052fe19a1bb435b65f6f5693a91e3ff5a9cdd"}
Oct 30 04:07:14.532383 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.530617    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"
Oct 30 04:07:14.533617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.533591    4877 generic.go:356] "Generic (PLEG): container finished" podID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerID="e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff" exitCode=0
Oct 30 04:07:14.534351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.533738    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" event={"ID":"ffc107c8-f93e-4124-979e-0a07d6f8df75","Type":"ContainerDied","Data":"e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff"}
Oct 30 04:07:14.534351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.533771    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf" event={"ID":"ffc107c8-f93e-4124-979e-0a07d6f8df75","Type":"ContainerDied","Data":"d3c84dfad88befae5d688f9817be5336af19228a995ece1fa4c7c6b24e40a681"}
Oct 30 04:07:14.534351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.533881    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"
Oct 30 04:07:14.546443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.543619    4877 scope.go:122] "RemoveContainer" containerID="d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"
Oct 30 04:07:14.562779 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.562753    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs\" (UniqueName: \"kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs\") pod \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") "
Oct 30 04:07:14.563003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.562974    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume\") pod \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\" (UID: \"1ff8c4ad-1da5-44ac-9f36-75dce02091c4\") "
Oct 30 04:07:14.563132 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563017    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm\") pod \"ffc107c8-f93e-4124-979e-0a07d6f8df75\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") "
Oct 30 04:07:14.563132 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563093    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh\" (UniqueName: \"kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh\") pod \"ffc107c8-f93e-4124-979e-0a07d6f8df75\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") "
Oct 30 04:07:14.563260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563158    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir\") pod \"ffc107c8-f93e-4124-979e-0a07d6f8df75\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") "
Oct 30 04:07:14.563317 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563283    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs\") pod \"ffc107c8-f93e-4124-979e-0a07d6f8df75\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") "
Oct 30 04:07:14.563376 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563352    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache\") pod \"ffc107c8-f93e-4124-979e-0a07d6f8df75\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") "
Oct 30 04:07:14.563475 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563408    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home\") pod \"ffc107c8-f93e-4124-979e-0a07d6f8df75\" (UID: \"ffc107c8-f93e-4124-979e-0a07d6f8df75\") "
Oct 30 04:07:14.563475 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563405    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume" pod "1ff8c4ad-1da5-44ac-9f36-75dce02091c4" (UID: "1ff8c4ad-1da5-44ac-9f36-75dce02091c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:14.563743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563718    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir" pod "ffc107c8-f93e-4124-979e-0a07d6f8df75" (UID: "ffc107c8-f93e-4124-979e-0a07d6f8df75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.563977 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.563952    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home" pod "ffc107c8-f93e-4124-979e-0a07d6f8df75" (UID: "ffc107c8-f93e-4124-979e-0a07d6f8df75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564067    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5k6n\" (UniqueName: \"kubernetes.io/projected/3ee1492c-d2c2-4e86-818b-514ac3368623-kube-api-access-p5k6n\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564099    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564115    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564132    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee1492c-d2c2-4e86-818b-514ac3368623-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564148    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564162    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75sn\" (UniqueName: \"kubernetes.io/projected/4f31e566-4b61-40b3-a685-4dd0c4bfce2b-kube-api-access-l75sn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564175    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee1492c-d2c2-4e86-818b-514ac3368623-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.564467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.564188    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.565285 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.565258    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache" pod "ffc107c8-f93e-4124-979e-0a07d6f8df75" (UID: "ffc107c8-f93e-4124-979e-0a07d6f8df75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.565700 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.565679    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:14.568584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.567983    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs" pod "ffc107c8-f93e-4124-979e-0a07d6f8df75" (UID: "ffc107c8-f93e-4124-979e-0a07d6f8df75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:14.568987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.568851    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh" pod "ffc107c8-f93e-4124-979e-0a07d6f8df75" (UID: "ffc107c8-f93e-4124-979e-0a07d6f8df75"). InnerVolumeSpecName "kube-api-access-m9knh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:14.572405 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.572363    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm" pod "ffc107c8-f93e-4124-979e-0a07d6f8df75" (UID: "ffc107c8-f93e-4124-979e-0a07d6f8df75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:14.574038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.574007    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs" pod "1ff8c4ad-1da5-44ac-9f36-75dce02091c4" (UID: "1ff8c4ad-1da5-44ac-9f36-75dce02091c4"). InnerVolumeSpecName "kube-api-access-vf9qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:14.640622 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.640591    4877 scope.go:122] "RemoveContainer" containerID="58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693"
Oct 30 04:07:14.641212 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:14.641185    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693\": container with ID starting with 58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693 not found: ID does not exist" containerID="58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693"
Oct 30 04:07:14.641325 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.641221    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693"} err="failed to get container status \"58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693\": rpc error: code = NotFound desc = could not find container \"58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693\": container with ID starting with 58f9f6f0a5d707e9a810f6c181ebf0ba883ac328b291196e9a21e4413fc52693 not found: ID does not exist"
Oct 30 04:07:14.641325 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.641240    4877 scope.go:122] "RemoveContainer" containerID="d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"
Oct 30 04:07:14.642022 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:14.641842    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c\": container with ID starting with d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c not found: ID does not exist" containerID="d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"
Oct 30 04:07:14.642022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.641881    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c"} err="failed to get container status \"d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c\": rpc error: code = NotFound desc = could not find container \"d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c\": container with ID starting with d47427b9041334c361a1abf99e831e63dcdc2f5d69b80c1b72029c3e0d7e740c not found: ID does not exist"
Oct 30 04:07:14.642022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.641906    4877 scope.go:122] "RemoveContainer" containerID="c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742"
Oct 30 04:07:14.667220 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667184    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9qs\" (UniqueName: \"kubernetes.io/projected/1ff8c4ad-1da5-44ac-9f36-75dce02091c4-kube-api-access-vf9qs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.667396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667224    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.667396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667241    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9knh\" (UniqueName: \"kubernetes.io/projected/ffc107c8-f93e-4124-979e-0a07d6f8df75-kube-api-access-m9knh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.667396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667256    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc107c8-f93e-4124-979e-0a07d6f8df75-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.667396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667271    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.667396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667284    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc107c8-f93e-4124-979e-0a07d6f8df75-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:14.667760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.667740    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"]
Oct 30 04:07:14.671987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.670741    4877 scope.go:122] "RemoveContainer" containerID="c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742"
Oct 30 04:07:14.671987 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:14.671294    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742\": container with ID starting with c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742 not found: ID does not exist" containerID="c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742"
Oct 30 04:07:14.671987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.671334    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742"} err="failed to get container status \"c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742\": rpc error: code = NotFound desc = could not find container \"c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742\": container with ID starting with c07d5b3086daf955f43f902ed0a8a8a388c60f5b89439654cff5e018f90c2742 not found: ID does not exist"
Oct 30 04:07:14.671987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.671362    4877 scope.go:122] "RemoveContainer" containerID="d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c"
Oct 30 04:07:14.674895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.674861    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/premium-w1-e44313-kserve-6f74d6f4b5-w75fv"]
Oct 30 04:07:14.693072 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.693013    4877 scope.go:122] "RemoveContainer" containerID="d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c"
Oct 30 04:07:14.694197 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:14.693578    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c\": container with ID starting with d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c not found: ID does not exist" containerID="d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c"
Oct 30 04:07:14.694197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.693613    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c"} err="failed to get container status \"d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c\": rpc error: code = NotFound desc = could not find container \"d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c\": container with ID starting with d364e799c58f3c367c3c8ca7a30c78c3b505fb95670fdc7f3cc1139f72e2673c not found: ID does not exist"
Oct 30 04:07:14.694197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.693639    4877 scope.go:122] "RemoveContainer" containerID="b9aac0be858f2db1b1170ebbdac052fe19a1bb435b65f6f5693a91e3ff5a9cdd"
Oct 30 04:07:14.702574 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.702516    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"]
Oct 30 04:07:14.709390 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.709287    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-rbztd"]
Oct 30 04:07:14.725971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.725950    4877 scope.go:122] "RemoveContainer" containerID="e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff"
Oct 30 04:07:14.736277 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.736252    4877 scope.go:122] "RemoveContainer" containerID="e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff"
Oct 30 04:07:14.736663 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:14.736643    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff\": container with ID starting with e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff not found: ID does not exist" containerID="e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff"
Oct 30 04:07:14.736786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.736749    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff"} err="failed to get container status \"e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff\": rpc error: code = NotFound desc = could not find container \"e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff\": container with ID starting with e4e515cb469837b0e2ecd48a663aba86409b74b9ceb7336e746b9795e4bb76ff not found: ID does not exist"
Oct 30 04:07:14.740908 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.740888    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"]
Oct 30 04:07:14.754332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.754307    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e44313-596b94c9b-v6z85"]
Oct 30 04:07:14.875587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.875561    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"]
Oct 30 04:07:14.886353 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.886284    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e44313-7f7d8f97b7-hnhf2"]
Oct 30 04:07:14.913109 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.913062    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"]
Oct 30 04:07:14.918921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.918885    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/simulator-w1-e44313-kserve-6f659d4b5b-kmvbf"]
Oct 30 04:07:14.927330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:14.927307    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"]
Oct 30 04:07:15.546775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.546739    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8" event={"ID":"e097a1b6-6520-4472-bddf-adfeb6171e79","Type":"ContainerStarted","Data":"7bdbca38976418cffa9a135af9e8f0b2a8d91589d09b64be7c9ed93a92d1b097"}
Oct 30 04:07:15.547244 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.546804    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8" event={"ID":"e097a1b6-6520-4472-bddf-adfeb6171e79","Type":"ContainerStarted","Data":"0d6d5e365fb3233bc25af7ef1c7b6f5bda094f2968ebcb865f73a48ffddab1af"}
Oct 30 04:07:15.585348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.585296    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8" podStartSLOduration=1.5852827600000001 podStartE2EDuration="1.58528276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:15.5799518 +0000 UTC m=+3864.008841219" watchObservedRunningTime="2026-10-30 04:07:15.58528276 +0000 UTC m=+3864.014172177"
Oct 30 04:07:15.691316 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.691281    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:07:15.691496 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.691344    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:07:15.702281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.702241    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff8c4ad-1da5-44ac-9f36-75dce02091c4" path="/var/lib/kubelet/pods/1ff8c4ad-1da5-44ac-9f36-75dce02091c4/volumes"
Oct 30 04:07:15.702821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.702795    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" path="/var/lib/kubelet/pods/3ee1492c-d2c2-4e86-818b-514ac3368623/volumes"
Oct 30 04:07:15.703261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.703239    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f31e566-4b61-40b3-a685-4dd0c4bfce2b" path="/var/lib/kubelet/pods/4f31e566-4b61-40b3-a685-4dd0c4bfce2b/volumes"
Oct 30 04:07:15.704104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.704075    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4348e6-070a-4199-952b-58649a0ef0f6" path="/var/lib/kubelet/pods/bf4348e6-070a-4199-952b-58649a0ef0f6/volumes"
Oct 30 04:07:15.704575 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:15.704558    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" path="/var/lib/kubelet/pods/ffc107c8-f93e-4124-979e-0a07d6f8df75/volumes"
Oct 30 04:07:16.236556 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:16.236468    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:07:16.557955 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:16.557920    4877 generic.go:356] "Generic (PLEG): container finished" podID="e097a1b6-6520-4472-bddf-adfeb6171e79" containerID="7bdbca38976418cffa9a135af9e8f0b2a8d91589d09b64be7c9ed93a92d1b097" exitCode=0
Oct 30 04:07:16.558367 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:16.557975    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8" event={"ID":"e097a1b6-6520-4472-bddf-adfeb6171e79","Type":"ContainerDied","Data":"7bdbca38976418cffa9a135af9e8f0b2a8d91589d09b64be7c9ed93a92d1b097"}
Oct 30 04:07:17.730996 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.730974    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:17.811143 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.811110    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca\") pod \"e097a1b6-6520-4472-bddf-adfeb6171e79\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") "
Oct 30 04:07:17.811353 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.811242    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca\") pod \"e097a1b6-6520-4472-bddf-adfeb6171e79\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") "
Oct 30 04:07:17.811456 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.811401    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp\" (UniqueName: \"kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp\") pod \"e097a1b6-6520-4472-bddf-adfeb6171e79\" (UID: \"e097a1b6-6520-4472-bddf-adfeb6171e79\") "
Oct 30 04:07:17.811909 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.811871    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca" pod "e097a1b6-6520-4472-bddf-adfeb6171e79" (UID: "e097a1b6-6520-4472-bddf-adfeb6171e79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:17.813560 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.813534    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca" pod "e097a1b6-6520-4472-bddf-adfeb6171e79" (UID: "e097a1b6-6520-4472-bddf-adfeb6171e79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:17.814154 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.814116    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp" pod "e097a1b6-6520-4472-bddf-adfeb6171e79" (UID: "e097a1b6-6520-4472-bddf-adfeb6171e79"). InnerVolumeSpecName "kube-api-access-fpkcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:17.913290 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.913251    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:17.913290 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.913288    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkcp\" (UniqueName: \"kubernetes.io/projected/e097a1b6-6520-4472-bddf-adfeb6171e79-kube-api-access-fpkcp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:17.913527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:17.913306    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e097a1b6-6520-4472-bddf-adfeb6171e79-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.581402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.581303    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8" event={"ID":"e097a1b6-6520-4472-bddf-adfeb6171e79","Type":"ContainerDied","Data":"0d6d5e365fb3233bc25af7ef1c7b6f5bda094f2968ebcb865f73a48ffddab1af"}
Oct 30 04:07:18.581402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.581365    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d6d5e365fb3233bc25af7ef1c7b6f5bda094f2968ebcb865f73a48ffddab1af"
Oct 30 04:07:18.581747 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.581454    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"
Oct 30 04:07:18.586098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.586059    4877 generic.go:356] "Generic (PLEG): container finished" podID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerID="53aecb49627b3294e058397fdbe7291cf11faa26dd47688d7cbb20313a34ba22" exitCode=0
Oct 30 04:07:18.586255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.586134    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" event={"ID":"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af","Type":"ContainerDied","Data":"53aecb49627b3294e058397fdbe7291cf11faa26dd47688d7cbb20313a34ba22"}
Oct 30 04:07:18.586255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.586183    4877 scope.go:122] "RemoveContainer" containerID="7ababe62034b7444763eb5f84f31411c767c61646d2a96d348dc19e308d487c2"
Oct 30 04:07:18.606870 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.606844    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 04:07:18.724793 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.724753    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.724987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.724869    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725001    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725113 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725076    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725182 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725122    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725182 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725167    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725288 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725232    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725286    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725328    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:18.725396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725342    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:18.725396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725337    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket\") pod \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\" (UID: \"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af\") "
Oct 30 04:07:18.725586 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725561    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:18.725973 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.725947    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:18.726201 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.726038    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.726201 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.726066    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.726201 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.726082    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.726399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.726320    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:18.727740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.727713    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "kube-api-access-7skdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:18.727885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.727866    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:18.727960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.727880    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:18.728052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.728027    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo" pod "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" (UID: "58abc3ac-e0c7-451d-8b8b-2f6e39ea89af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:07:18.826998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.826959    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.827547 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.827007    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.827547 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.827023    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.827547 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.827032    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7skdw\" (UniqueName: \"kubernetes.io/projected/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-kube-api-access-7skdw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.827547 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.827046    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:18.827547 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:18.827057    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:19.596078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:19.596042    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"
Oct 30 04:07:19.596578 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:19.596074    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62" event={"ID":"58abc3ac-e0c7-451d-8b8b-2f6e39ea89af","Type":"ContainerDied","Data":"267b505f7e7b4898fd822e74577a6610c81f081bff2cc07cc9661b47188dab6c"}
Oct 30 04:07:19.596578 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:19.596129    4877 scope.go:122] "RemoveContainer" containerID="53aecb49627b3294e058397fdbe7291cf11faa26dd47688d7cbb20313a34ba22"
Oct 30 04:07:19.717891 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:19.717849    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"]
Oct 30 04:07:19.729949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:19.729907    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-817531-openshift-default-5b6ff899bc-clh62"]
Oct 30 04:07:21.699165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:21.699127    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" path="/var/lib/kubelet/pods/58abc3ac-e0c7-451d-8b8b-2f6e39ea89af/volumes"
Oct 30 04:07:24.541949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:24.541913    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"]
Oct 30 04:07:24.542634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:24.542167    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy" containerID="cri-o://557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91" gracePeriod=30
Oct 30 04:07:25.358387 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:25.358346    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"]
Oct 30 04:07:25.394205 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:25.394171    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"]
Oct 30 04:07:26.040966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.040926    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"]
Oct 30 04:07:26.041554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041537    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1ff8c4ad-1da5-44ac-9f36-75dce02091c4" containerName="payload-pre-processing"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041556    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff8c4ad-1da5-44ac-9f36-75dce02091c4" containerName="payload-pre-processing"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041585    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041590    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041597    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041602    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041610    4877 cpu_manager.go:414] "containerMap: removing container" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041614    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy"
Oct 30 04:07:26.041620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041622    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e097a1b6-6520-4472-bddf-adfeb6171e79" containerName="revoke-keys"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041626    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e097a1b6-6520-4472-bddf-adfeb6171e79" containerName="revoke-keys"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041652    4877 cpu_manager.go:414] "containerMap: removing container" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041656    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041668    4877 cpu_manager.go:414] "containerMap: removing container" podUID="4f31e566-4b61-40b3-a685-4dd0c4bfce2b" containerName="payload-processing"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041672    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f31e566-4b61-40b3-a685-4dd0c4bfce2b" containerName="payload-processing"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041749    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041755    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e097a1b6-6520-4472-bddf-adfeb6171e79" containerName="revoke-keys"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041766    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="58abc3ac-e0c7-451d-8b8b-2f6e39ea89af" containerName="istio-proxy"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041772    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f31e566-4b61-40b3-a685-4dd0c4bfce2b" containerName="payload-processing"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041778    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff8c4ad-1da5-44ac-9f36-75dce02091c4" containerName="payload-pre-processing"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041788    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc107c8-f93e-4124-979e-0a07d6f8df75" containerName="main"
Oct 30 04:07:26.041885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.041793    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee1492c-d2c2-4e86-818b-514ac3368623" containerName="main"
Oct 30 04:07:26.047002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.046983    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.059844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.059824    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:07:26.059957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.059886    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:07:26.065913 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.065888    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"]
Oct 30 04:07:26.104761 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.104720    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddpj6\" (UniqueName: \"kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.104983 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.104769    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.105066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.105000    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.206404 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.206359    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.206874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.206850    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddpj6\" (UniqueName: \"kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.207105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.207086    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.207254 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.207226    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.232051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.232019    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.233353 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.233290    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddpj6\" (UniqueName: \"kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6\") pod \"maas-api-revoke-sub-b6073175-2jbgq\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.361358 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.361266    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:26.689953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.689926    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"]
Oct 30 04:07:26.693516 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:07:26.693483    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963c2032_7eb9_4e3c_aa24_9dc979d80371.slice/crio-dadd2b66689da8a158bb679a56a1434f4bcbfcaea7934706547cfe21f108cb52 WatchSource:0}: Error finding container dadd2b66689da8a158bb679a56a1434f4bcbfcaea7934706547cfe21f108cb52: Status 404 returned error can't find the container with id dadd2b66689da8a158bb679a56a1434f4bcbfcaea7934706547cfe21f108cb52
Oct 30 04:07:26.787276 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:26.787238    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq" event={"ID":"963c2032-7eb9-4e3c-aa24-9dc979d80371","Type":"ContainerStarted","Data":"dadd2b66689da8a158bb679a56a1434f4bcbfcaea7934706547cfe21f108cb52"}
Oct 30 04:07:27.799044 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:27.799009    4877 generic.go:356] "Generic (PLEG): container finished" podID="963c2032-7eb9-4e3c-aa24-9dc979d80371" containerID="c8df4e6049c8aa743effaf635914dddfc376b62df76b367756a1aaf227cfccc6" exitCode=0
Oct 30 04:07:27.799580 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:27.799092    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq" event={"ID":"963c2032-7eb9-4e3c-aa24-9dc979d80371","Type":"ContainerDied","Data":"c8df4e6049c8aa743effaf635914dddfc376b62df76b367756a1aaf227cfccc6"}
Oct 30 04:07:28.997888 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:28.997859    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:29.068645 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.068557    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca\") pod \"963c2032-7eb9-4e3c-aa24-9dc979d80371\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") "
Oct 30 04:07:29.068826 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.068646    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca\") pod \"963c2032-7eb9-4e3c-aa24-9dc979d80371\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") "
Oct 30 04:07:29.068826 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.068725    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6\" (UniqueName: \"kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6\") pod \"963c2032-7eb9-4e3c-aa24-9dc979d80371\" (UID: \"963c2032-7eb9-4e3c-aa24-9dc979d80371\") "
Oct 30 04:07:29.069160 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.069130    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca" pod "963c2032-7eb9-4e3c-aa24-9dc979d80371" (UID: "963c2032-7eb9-4e3c-aa24-9dc979d80371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:29.069303 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.069278    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.071062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.071035    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca" pod "963c2032-7eb9-4e3c-aa24-9dc979d80371" (UID: "963c2032-7eb9-4e3c-aa24-9dc979d80371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:29.071562 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.071544    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6" pod "963c2032-7eb9-4e3c-aa24-9dc979d80371" (UID: "963c2032-7eb9-4e3c-aa24-9dc979d80371"). InnerVolumeSpecName "kube-api-access-ddpj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:29.170658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.170621    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963c2032-7eb9-4e3c-aa24-9dc979d80371-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.170658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.170653    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddpj6\" (UniqueName: \"kubernetes.io/projected/963c2032-7eb9-4e3c-aa24-9dc979d80371-kube-api-access-ddpj6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.809197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.809162    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 04:07:29.835398 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.835065    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"
Oct 30 04:07:29.835398 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.835098    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq" event={"ID":"963c2032-7eb9-4e3c-aa24-9dc979d80371","Type":"ContainerDied","Data":"dadd2b66689da8a158bb679a56a1434f4bcbfcaea7934706547cfe21f108cb52"}
Oct 30 04:07:29.835398 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.835162    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dadd2b66689da8a158bb679a56a1434f4bcbfcaea7934706547cfe21f108cb52"
Oct 30 04:07:29.838481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.837388    4877 generic.go:356] "Generic (PLEG): container finished" podID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerID="557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91" exitCode=0
Oct 30 04:07:29.838481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.837622    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" event={"ID":"97fa1ab1-f125-46d3-9836-9bc979fe9cd3","Type":"ContainerDied","Data":"557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91"}
Oct 30 04:07:29.838481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.837671    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"
Oct 30 04:07:29.838481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.837685    4877 scope.go:122] "RemoveContainer" containerID="557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91"
Oct 30 04:07:29.838481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.837669    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7" event={"ID":"97fa1ab1-f125-46d3-9836-9bc979fe9cd3","Type":"ContainerDied","Data":"4c373c148f1004e39aea3a416408a6b8584a028ba816a1a4c77b60276ba47b77"}
Oct 30 04:07:29.866703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.865754    4877 scope.go:122] "RemoveContainer" containerID="c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"
Oct 30 04:07:29.884003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.883979    4877 scope.go:122] "RemoveContainer" containerID="557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91"
Oct 30 04:07:29.885140 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:29.885109    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91\": container with ID starting with 557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91 not found: ID does not exist" containerID="557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91"
Oct 30 04:07:29.885332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.885151    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91"} err="failed to get container status \"557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91\": rpc error: code = NotFound desc = could not find container \"557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91\": container with ID starting with 557ed66e75c30972631aa8fa95f4d01e615c64664e99fd39f5ff53621788be91 not found: ID does not exist"
Oct 30 04:07:29.885332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.885175    4877 scope.go:122] "RemoveContainer" containerID="c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"
Oct 30 04:07:29.885550 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:29.885525    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446\": container with ID starting with c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446 not found: ID does not exist" containerID="c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"
Oct 30 04:07:29.885620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.885558    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446"} err="failed to get container status \"c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446\": rpc error: code = NotFound desc = could not find container \"c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446\": container with ID starting with c7ed149f69733b474e2b63ddea01932035a4d7b82a2a19b4200707e684e96446 not found: ID does not exist"
Oct 30 04:07:29.890712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890460    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.890712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890510    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.890712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890591    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.890712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890680    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.891011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890757    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.891011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890815    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.891011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890916    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.891011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.890969    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.891226 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.891031    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb\") pod \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\" (UID: \"97fa1ab1-f125-46d3-9836-9bc979fe9cd3\") "
Oct 30 04:07:29.891718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.891536    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:29.892148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.891864    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:29.892479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.892278    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:29.892877 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.892603    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:29.893672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.893637    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:29.894989 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.894965    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:29.895169 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.895150    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:29.896816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.896789    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "kube-api-access-69tmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:29.898104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.898075    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo" pod "97fa1ab1-f125-46d3-9836-9bc979fe9cd3" (UID: "97fa1ab1-f125-46d3-9836-9bc979fe9cd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:07:29.993105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993073    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993101    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993110    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69tmb\" (UniqueName: \"kubernetes.io/projected/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-kube-api-access-69tmb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993119    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993129    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993137    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993145    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993154    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:29.993306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:29.993163    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fa1ab1-f125-46d3-9836-9bc979fe9cd3-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:30.186837 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:30.186799    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"]
Oct 30 04:07:30.191255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:30.191223    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-78hb7"]
Oct 30 04:07:30.358902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:30.358853    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerName="payload-processing" containerID="cri-o://a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94" gracePeriod=25
Oct 30 04:07:30.395249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:30.395208    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" containerName="payload-pre-processing" containerID="cri-o://cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418" gracePeriod=25
Oct 30 04:07:31.607764 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:31.607731    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"]
Oct 30 04:07:31.701168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:31.701131    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" path="/var/lib/kubelet/pods/97fa1ab1-f125-46d3-9836-9bc979fe9cd3/volumes"
Oct 30 04:07:36.608967 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:36.608906    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" podUID="51dd22c0-d892-4843-816a-71abd0d92556" containerName="payload-pre-processing" containerID="cri-o://f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19" gracePeriod=25
Oct 30 04:07:38.344709 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:38.344669    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:38.345199 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:38.344734    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:41.220654 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:41.220611    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"]
Oct 30 04:07:41.221161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:41.220885    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy" containerID="cri-o://0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef" gracePeriod=30
Oct 30 04:07:46.230674 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:46.230610    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:07:46.530645 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.530615    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 04:07:46.550495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550461    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550581    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550619    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550643    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550661    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550684    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550727    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550821    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.550998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.550845    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb\") pod \"3d3bd079-fc0a-4671-b826-b2369583768d\" (UID: \"3d3bd079-fc0a-4671-b826-b2369583768d\") "
Oct 30 04:07:46.551881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.551819    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:46.552023 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.551906    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:46.552193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.552164    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:46.552326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.552274    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:46.552505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.552465    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:46.553978 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.553952    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "kube-api-access-hbqgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:46.554102 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.554051    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:46.554384 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.554361    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:07:46.555044 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.555019    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy" pod "3d3bd079-fc0a-4671-b826-b2369583768d" (UID: "3d3bd079-fc0a-4671-b826-b2369583768d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:46.599001 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.598959    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" podUID="51dd22c0-d892-4843-816a-71abd0d92556" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:46.652637 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652603    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652637 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652632    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652637 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652643    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbqgb\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-kube-api-access-hbqgb\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652652    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652662    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652670    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d3bd079-fc0a-4671-b826-b2369583768d-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652678    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d3bd079-fc0a-4671-b826-b2369583768d-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652686    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d3bd079-fc0a-4671-b826-b2369583768d-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.652881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.652694    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d3bd079-fc0a-4671-b826-b2369583768d-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:46.985868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.985822    4877 generic.go:356] "Generic (PLEG): container finished" podID="3d3bd079-fc0a-4671-b826-b2369583768d" containerID="0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef" exitCode=0
Oct 30 04:07:46.986062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.985870    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"
Oct 30 04:07:46.986062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.985890    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" event={"ID":"3d3bd079-fc0a-4671-b826-b2369583768d","Type":"ContainerDied","Data":"0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef"}
Oct 30 04:07:46.986062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.985952    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x" event={"ID":"3d3bd079-fc0a-4671-b826-b2369583768d","Type":"ContainerDied","Data":"213c7933e2e81cac05135e9ae79c161a78ca234cd9b460a99490c63f2e528e09"}
Oct 30 04:07:46.986062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:46.985984    4877 scope.go:122] "RemoveContainer" containerID="0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef"
Oct 30 04:07:47.000738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.000717    4877 scope.go:122] "RemoveContainer" containerID="8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"
Oct 30 04:07:47.011369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.011345    4877 scope.go:122] "RemoveContainer" containerID="0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef"
Oct 30 04:07:47.011746 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:47.011717    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef\": container with ID starting with 0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef not found: ID does not exist" containerID="0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef"
Oct 30 04:07:47.011862 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.011759    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef"} err="failed to get container status \"0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef\": rpc error: code = NotFound desc = could not find container \"0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef\": container with ID starting with 0075b10e1018bc555aa0914c7fbccc7bbaab8901f9c7afeee89fa7ac77b75bef not found: ID does not exist"
Oct 30 04:07:47.011862 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.011780    4877 scope.go:122] "RemoveContainer" containerID="8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"
Oct 30 04:07:47.012146 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:47.012118    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20\": container with ID starting with 8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20 not found: ID does not exist" containerID="8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"
Oct 30 04:07:47.012267 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.012154    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20"} err="failed to get container status \"8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20\": rpc error: code = NotFound desc = could not find container \"8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20\": container with ID starting with 8fce79775fc997e4c6c7316aaf600ad29434b96869fc6f3388b63fea7ff0fc20 not found: ID does not exist"
Oct 30 04:07:47.034504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.034473    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"]
Oct 30 04:07:47.045789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.045760    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-fdm9x"]
Oct 30 04:07:47.698800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:47.698768    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" path="/var/lib/kubelet/pods/3d3bd079-fc0a-4671-b826-b2369583768d/volumes"
Oct 30 04:07:48.344573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:48.344530    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:48.344763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:48.344568    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:50.229613 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.229568    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"]
Oct 30 04:07:50.267229 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.267190    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"]
Oct 30 04:07:50.724096 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.724012    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 04:07:50.749814 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.749789    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.847919    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume\") pod \"3eae2566-d17c-4e70-96bc-c417fa401866\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") "
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.848001    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume\") pod \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") "
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.848038    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr\" (UniqueName: \"kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr\") pod \"3eae2566-d17c-4e70-96bc-c417fa401866\" (UID: \"3eae2566-d17c-4e70-96bc-c417fa401866\") "
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.848158    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp\" (UniqueName: \"kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp\") pod \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\" (UID: \"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286\") "
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.854716    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr" pod "3eae2566-d17c-4e70-96bc-c417fa401866" (UID: "3eae2566-d17c-4e70-96bc-c417fa401866"). InnerVolumeSpecName "kube-api-access-ccrvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.855171    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume" pod "3eae2566-d17c-4e70-96bc-c417fa401866" (UID: "3eae2566-d17c-4e70-96bc-c417fa401866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:50.856453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.855863    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume" pod "a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" (UID: "a62ea3ee-bba8-42d1-98b7-8bb07ee9c286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:50.864442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.861050    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp" pod "a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" (UID: "a62ea3ee-bba8-42d1-98b7-8bb07ee9c286"). InnerVolumeSpecName "kube-api-access-9jkfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:50.953232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.953195    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:50.953232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.953230    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrvr\" (UniqueName: \"kubernetes.io/projected/3eae2566-d17c-4e70-96bc-c417fa401866-kube-api-access-ccrvr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:50.953232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.953239    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkfp\" (UniqueName: \"kubernetes.io/projected/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286-kube-api-access-9jkfp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:50.953587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:50.953247    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eae2566-d17c-4e70-96bc-c417fa401866-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:51.079333 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.079294    4877 generic.go:356] "Generic (PLEG): container finished" podID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerID="a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94" exitCode=0
Oct 30 04:07:51.079566 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.079380    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" event={"ID":"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286","Type":"ContainerDied","Data":"a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94"}
Oct 30 04:07:51.079566 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.079470    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j" event={"ID":"a62ea3ee-bba8-42d1-98b7-8bb07ee9c286","Type":"ContainerDied","Data":"93047dd775fbf6b1757b363d0f9cffbf424b189c287643b3da6282fbcc875998"}
Oct 30 04:07:51.079566 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.079498    4877 scope.go:122] "RemoveContainer" containerID="a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94"
Oct 30 04:07:51.079763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.079628    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"
Oct 30 04:07:51.087605 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.087263    4877 generic.go:356] "Generic (PLEG): container finished" podID="3eae2566-d17c-4e70-96bc-c417fa401866" containerID="cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418" exitCode=0
Oct 30 04:07:51.087605 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.087313    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"
Oct 30 04:07:51.087605 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.087337    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" event={"ID":"3eae2566-d17c-4e70-96bc-c417fa401866","Type":"ContainerDied","Data":"cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418"}
Oct 30 04:07:51.087605 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.087401    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz" event={"ID":"3eae2566-d17c-4e70-96bc-c417fa401866","Type":"ContainerDied","Data":"8060515aa20d5594ddba0ad293e345343b3abf4ef2dff9cd4416ab36718bfa5e"}
Oct 30 04:07:51.104781 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.104758    4877 scope.go:122] "RemoveContainer" containerID="a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94"
Oct 30 04:07:51.105131 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:51.105108    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94\": container with ID starting with a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94 not found: ID does not exist" containerID="a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94"
Oct 30 04:07:51.105222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.105143    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94"} err="failed to get container status \"a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94\": rpc error: code = NotFound desc = could not find container \"a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94\": container with ID starting with a00d0351ab15c14cfb75f7f783afcc6088f854c56ba3f7b9614d32da8ebeaa94 not found: ID does not exist"
Oct 30 04:07:51.105222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.105168    4877 scope.go:122] "RemoveContainer" containerID="cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418"
Oct 30 04:07:51.118325 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.118302    4877 scope.go:122] "RemoveContainer" containerID="cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418"
Oct 30 04:07:51.118690 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:51.118672    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418\": container with ID starting with cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418 not found: ID does not exist" containerID="cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418"
Oct 30 04:07:51.118777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.118698    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418"} err="failed to get container status \"cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418\": rpc error: code = NotFound desc = could not find container \"cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418\": container with ID starting with cc1e5651da3abc22a0294ccd6145fa58cd0626397d2fc6f53de92d3f31ec6418 not found: ID does not exist"
Oct 30 04:07:51.126195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.126173    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"]
Oct 30 04:07:51.135203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.135180    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a9c9b6-5c46996b86-fjh4j"]
Oct 30 04:07:51.170159 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.170138    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"]
Oct 30 04:07:51.174053 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.174031    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a9c9b6-7d594f9c4c-7nzbz"]
Oct 30 04:07:51.701242 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.701163    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" path="/var/lib/kubelet/pods/3eae2566-d17c-4e70-96bc-c417fa401866/volumes"
Oct 30 04:07:51.701626 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:51.701578    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" path="/var/lib/kubelet/pods/a62ea3ee-bba8-42d1-98b7-8bb07ee9c286/volumes"
Oct 30 04:07:53.024705 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:53.024671    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:07:53.049831 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:53.049798    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:07:55.230721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:55.230668    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerName="payload-processing" containerID="cri-o://4f497894e4b49cec3818712de391b3ef576c207b4ec19efb6e893673dbec20a8" gracePeriod=25
Oct 30 04:07:55.267952 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:55.267907    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerName="payload-pre-processing" containerID="cri-o://6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613" gracePeriod=25
Oct 30 04:07:55.932492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:55.932457    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"]
Oct 30 04:07:55.933079 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:55.932794    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy" containerID="cri-o://70ba340a460928d45f420220846f8f48fc141b40d1fd736b4a61672479ac1c3d" gracePeriod=30
Oct 30 04:07:56.599081 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.599038    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" podUID="51dd22c0-d892-4843-816a-71abd0d92556" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:07:56.904263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.904232    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 04:07:56.916464 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.916417    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"]
Oct 30 04:07:56.917002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.916985    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy"
Oct 30 04:07:56.917002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917001    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917017    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerName="payload-processing"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917023    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerName="payload-processing"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917036    4877 cpu_manager.go:414] "containerMap: removing container" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917041    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917054    4877 cpu_manager.go:414] "containerMap: removing container" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917058    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917065    4877 cpu_manager.go:414] "containerMap: removing container" podUID="963c2032-7eb9-4e3c-aa24-9dc979d80371" containerName="revoke-keys"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917069    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963c2032-7eb9-4e3c-aa24-9dc979d80371" containerName="revoke-keys"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917086    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" containerName="payload-pre-processing"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917092    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" containerName="payload-pre-processing"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917113    4877 cpu_manager.go:414] "containerMap: removing container" podUID="51dd22c0-d892-4843-816a-71abd0d92556" containerName="payload-pre-processing"
Oct 30 04:07:56.917141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917118    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51dd22c0-d892-4843-816a-71abd0d92556" containerName="payload-pre-processing"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917211    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917221    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fa1ab1-f125-46d3-9836-9bc979fe9cd3" containerName="istio-proxy"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917229    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="963c2032-7eb9-4e3c-aa24-9dc979d80371" containerName="revoke-keys"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917236    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917243    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eae2566-d17c-4e70-96bc-c417fa401866" containerName="payload-pre-processing"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917251    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="51dd22c0-d892-4843-816a-71abd0d92556" containerName="payload-pre-processing"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917258    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy"
Oct 30 04:07:56.917854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.917265    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62ea3ee-bba8-42d1-98b7-8bb07ee9c286" containerName="payload-processing"
Oct 30 04:07:56.920630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.920611    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:56.936756 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.936731    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:07:56.942006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.940818    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:56.942006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.940931    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:56.942006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.941007    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bjm\" (UniqueName: \"kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:56.942360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:56.942294    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"]
Oct 30 04:07:57.020558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.020475    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:07:57.042025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.041976    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr\" (UniqueName: \"kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr\") pod \"51dd22c0-d892-4843-816a-71abd0d92556\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") "
Oct 30 04:07:57.042197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.042170    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume\") pod \"51dd22c0-d892-4843-816a-71abd0d92556\" (UID: \"51dd22c0-d892-4843-816a-71abd0d92556\") "
Oct 30 04:07:57.042433 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.042396    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.042588 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.042482    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.042588 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.042544    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bjm\" (UniqueName: \"kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.042588 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.042541    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume" pod "51dd22c0-d892-4843-816a-71abd0d92556" (UID: "51dd22c0-d892-4843-816a-71abd0d92556"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:57.042801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.042771    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51dd22c0-d892-4843-816a-71abd0d92556-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.043195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.043174    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.044207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.044187    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr" pod "51dd22c0-d892-4843-816a-71abd0d92556" (UID: "51dd22c0-d892-4843-816a-71abd0d92556"). InnerVolumeSpecName "kube-api-access-q5sqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:57.044737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.044719    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.059221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.059191    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bjm\" (UniqueName: \"kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm\") pod \"maas-api-revoke-sub-284a7d5b-b5hxv\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.143668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.143587    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5sqr\" (UniqueName: \"kubernetes.io/projected/51dd22c0-d892-4843-816a-71abd0d92556-kube-api-access-q5sqr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.144777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.144750    4877 generic.go:356] "Generic (PLEG): container finished" podID="51dd22c0-d892-4843-816a-71abd0d92556" containerID="f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19" exitCode=0
Oct 30 04:07:57.144890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.144811    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"
Oct 30 04:07:57.144890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.144831    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" event={"ID":"51dd22c0-d892-4843-816a-71abd0d92556","Type":"ContainerDied","Data":"f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19"}
Oct 30 04:07:57.145019 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.144924    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr" event={"ID":"51dd22c0-d892-4843-816a-71abd0d92556","Type":"ContainerDied","Data":"fe520ee22381bad60d5a5e916c820a3d1a72683190091335526136bb387e6ef9"}
Oct 30 04:07:57.145019 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.144959    4877 scope.go:122] "RemoveContainer" containerID="f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19"
Oct 30 04:07:57.156021 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.156003    4877 scope.go:122] "RemoveContainer" containerID="f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19"
Oct 30 04:07:57.156285 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:07:57.156264    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19\": container with ID starting with f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19 not found: ID does not exist" containerID="f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19"
Oct 30 04:07:57.156333 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.156293    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19"} err="failed to get container status \"f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19\": rpc error: code = NotFound desc = could not find container \"f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19\": container with ID starting with f0f15847db2d02b77861de95b3a59150a318472e2ca02531c1dd39292aeb6f19 not found: ID does not exist"
Oct 30 04:07:57.192333 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.192308    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"]
Oct 30 04:07:57.196087 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.196063    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e1a0c2-6d4f598895-nxnwr"]
Oct 30 04:07:57.234861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.234836    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:07:57.314780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.314751    4877 scope.go:122] "RemoveContainer" containerID="b6c4b00a2f082a27b1e78a68ea517077fce26e568cce8a0a54c97bdfd133e271"
Oct 30 04:07:57.327798 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.327755    4877 scope.go:122] "RemoveContainer" containerID="eb14208de003f15a14cb7195fc543177fd9fc0da6e5ccda20ba81950c60b0347"
Oct 30 04:07:57.581447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.578495    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"]
Oct 30 04:07:57.715459 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:57.715405    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51dd22c0-d892-4843-816a-71abd0d92556" path="/var/lib/kubelet/pods/51dd22c0-d892-4843-816a-71abd0d92556/volumes"
Oct 30 04:07:58.030712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:58.030678    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"]
Oct 30 04:07:58.031771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:58.030942    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy" containerID="cri-o://1afb8d12561ffe1d7e1f7c70a4d168cfd2b7d3574a801da3565fe47c75f4be8e" gracePeriod=30
Oct 30 04:07:58.189101 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:58.189063    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv" event={"ID":"ec98d1d5-a261-472c-b639-5ce3002206c9","Type":"ContainerStarted","Data":"bc13d6b7a3ab19b4ec8a96956bdce459260ee126b0befb2086beb50ba4de2e48"}
Oct 30 04:07:58.190380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:58.190355    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv" event={"ID":"ec98d1d5-a261-472c-b639-5ce3002206c9","Type":"ContainerStarted","Data":"bc595b9653f362de32726eaabea30ef76e277169a169f46adc5275f4b5d1e8a6"}
Oct 30 04:07:58.225863 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:58.225796    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv" podStartSLOduration=2.225777929 podStartE2EDuration="2.225777929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:58.224031287 +0000 UTC m=+3906.652920708" watchObservedRunningTime="2026-10-30 04:07:58.225777929 +0000 UTC m=+3906.654667340"
Oct 30 04:07:59.198113 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:59.198074    4877 generic.go:356] "Generic (PLEG): container finished" podID="ec98d1d5-a261-472c-b639-5ce3002206c9" containerID="bc13d6b7a3ab19b4ec8a96956bdce459260ee126b0befb2086beb50ba4de2e48" exitCode=0
Oct 30 04:07:59.198539 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:07:59.198188    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv" event={"ID":"ec98d1d5-a261-472c-b639-5ce3002206c9","Type":"ContainerDied","Data":"bc13d6b7a3ab19b4ec8a96956bdce459260ee126b0befb2086beb50ba4de2e48"}
Oct 30 04:08:00.378672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.378643    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:08:00.396548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.396519    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca\") pod \"ec98d1d5-a261-472c-b639-5ce3002206c9\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") "
Oct 30 04:08:00.396679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.396591    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm\" (UniqueName: \"kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm\") pod \"ec98d1d5-a261-472c-b639-5ce3002206c9\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") "
Oct 30 04:08:00.396679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.396664    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca\") pod \"ec98d1d5-a261-472c-b639-5ce3002206c9\" (UID: \"ec98d1d5-a261-472c-b639-5ce3002206c9\") "
Oct 30 04:08:00.397024 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.397001    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca" pod "ec98d1d5-a261-472c-b639-5ce3002206c9" (UID: "ec98d1d5-a261-472c-b639-5ce3002206c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:00.397113 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.397081    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:00.398703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.398678    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm" pod "ec98d1d5-a261-472c-b639-5ce3002206c9" (UID: "ec98d1d5-a261-472c-b639-5ce3002206c9"). InnerVolumeSpecName "kube-api-access-42bjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:00.398793 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.398749    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca" pod "ec98d1d5-a261-472c-b639-5ce3002206c9" (UID: "ec98d1d5-a261-472c-b639-5ce3002206c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:00.497854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.497775    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec98d1d5-a261-472c-b639-5ce3002206c9-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:00.497854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:00.497808    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42bjm\" (UniqueName: \"kubernetes.io/projected/ec98d1d5-a261-472c-b639-5ce3002206c9-kube-api-access-42bjm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.217175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.217063    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv" event={"ID":"ec98d1d5-a261-472c-b639-5ce3002206c9","Type":"ContainerDied","Data":"bc595b9653f362de32726eaabea30ef76e277169a169f46adc5275f4b5d1e8a6"}
Oct 30 04:08:01.217175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.217129    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc595b9653f362de32726eaabea30ef76e277169a169f46adc5275f4b5d1e8a6"
Oct 30 04:08:01.221261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.217206    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"
Oct 30 04:08:01.225286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.225046    4877 generic.go:356] "Generic (PLEG): container finished" podID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerID="70ba340a460928d45f420220846f8f48fc141b40d1fd736b4a61672479ac1c3d" exitCode=0
Oct 30 04:08:01.225286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.225141    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" event={"ID":"e26db8af-a6b2-412d-87a6-e733209c41f2","Type":"ContainerDied","Data":"70ba340a460928d45f420220846f8f48fc141b40d1fd736b4a61672479ac1c3d"}
Oct 30 04:08:01.234458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.234436    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:08:01.307740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.307700    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.307929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.307816    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.307929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.307887    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.308066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.308000    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.308066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.308030    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.308066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.308061    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.308233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.308092    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.308233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.308162    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.308233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.308206    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn\") pod \"e26db8af-a6b2-412d-87a6-e733209c41f2\" (UID: \"e26db8af-a6b2-412d-87a6-e733209c41f2\") "
Oct 30 04:08:01.310256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.309413    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:01.310256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.309733    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:01.310256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.310219    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:01.310800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.310594    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:01.310800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.310770    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:01.313652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.313614    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:01.314683 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.314640    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "kube-api-access-g2qdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:01.318470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.315412    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:08:01.318470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.316697    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token" pod "e26db8af-a6b2-412d-87a6-e733209c41f2" (UID: "e26db8af-a6b2-412d-87a6-e733209c41f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:01.410222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410189    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410222    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qdn\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-kube-api-access-g2qdn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410236    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410245    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410254    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410262    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410270    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26db8af-a6b2-412d-87a6-e733209c41f2-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410279    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26db8af-a6b2-412d-87a6-e733209c41f2-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.410643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.410290    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26db8af-a6b2-412d-87a6-e733209c41f2-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:01.420903 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.420855    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"]
Oct 30 04:08:01.425504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.425473    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"]
Oct 30 04:08:01.434324 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:01.434299    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"]
Oct 30 04:08:02.234672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.234646    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"
Oct 30 04:08:02.234863 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.234689    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh" event={"ID":"e26db8af-a6b2-412d-87a6-e733209c41f2","Type":"ContainerDied","Data":"432d7bf730afd5bd9121e31ac86791e6dd586af338235efe538b75353e8576c0"}
Oct 30 04:08:02.234863 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.234757    4877 scope.go:122] "RemoveContainer" containerID="70ba340a460928d45f420220846f8f48fc141b40d1fd736b4a61672479ac1c3d"
Oct 30 04:08:02.291892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.291858    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"]
Oct 30 04:08:02.304095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.304066    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-bngsh"]
Oct 30 04:08:02.348192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.348153    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:02.348373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:02.348198    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:03.247204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.247171    4877 generic.go:356] "Generic (PLEG): container finished" podID="fa525320-3c49-4e12-badd-234109092779" containerID="1afb8d12561ffe1d7e1f7c70a4d168cfd2b7d3574a801da3565fe47c75f4be8e" exitCode=0
Oct 30 04:08:03.247737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.247236    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" event={"ID":"fa525320-3c49-4e12-badd-234109092779","Type":"ContainerDied","Data":"1afb8d12561ffe1d7e1f7c70a4d168cfd2b7d3574a801da3565fe47c75f4be8e"}
Oct 30 04:08:03.247737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.247303    4877 scope.go:122] "RemoveContainer" containerID="ee8745bc46f50dc0f0b4ac14178aa552c3051b23aca475abed55cc38ea1a6f90"
Oct 30 04:08:03.309510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.309412    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 04:08:03.332223 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332187    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332231    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332278    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332367    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332639 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332496    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332639 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332533    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332639 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332578    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332639 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332623    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332634    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:03.332868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332673    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data\") pod \"fa525320-3c49-4e12-badd-234109092779\" (UID: \"fa525320-3c49-4e12-badd-234109092779\") "
Oct 30 04:08:03.332868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332718    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:03.332868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.332804    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:03.333082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333024    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:03.333082 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333028    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:03.333281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333263    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.333354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333285    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa525320-3c49-4e12-badd-234109092779-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.333354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333300    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.333354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333312    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.333354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.333325    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.335053 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.335027    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:03.335181 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.335128    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:03.335181 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.335141    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:08:03.335181 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.335147    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l" pod "fa525320-3c49-4e12-badd-234109092779" (UID: "fa525320-3c49-4e12-badd-234109092779"). InnerVolumeSpecName "kube-api-access-lpr8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:03.434278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.434245    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpr8l\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-kube-api-access-lpr8l\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.434278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.434275    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa525320-3c49-4e12-badd-234109092779-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.434278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.434286    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa525320-3c49-4e12-badd-234109092779-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.434581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.434297    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa525320-3c49-4e12-badd-234109092779-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:03.699597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:03.699469    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" path="/var/lib/kubelet/pods/e26db8af-a6b2-412d-87a6-e733209c41f2/volumes"
Oct 30 04:08:04.260020 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:04.259987    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"
Oct 30 04:08:04.260020 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:04.260008    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j" event={"ID":"fa525320-3c49-4e12-badd-234109092779","Type":"ContainerDied","Data":"1194f52b3092704720774074b4d8d7ed7eb531177ebd20a52ef4645cab2dc38f"}
Oct 30 04:08:04.260552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:04.260101    4877 scope.go:122] "RemoveContainer" containerID="1afb8d12561ffe1d7e1f7c70a4d168cfd2b7d3574a801da3565fe47c75f4be8e"
Oct 30 04:08:04.322818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:04.322774    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"]
Oct 30 04:08:04.330825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:04.330798    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-qld8j"]
Oct 30 04:08:05.700206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:05.700173    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa525320-3c49-4e12-badd-234109092779" path="/var/lib/kubelet/pods/fa525320-3c49-4e12-badd-234109092779/volumes"
Oct 30 04:08:12.259361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259321    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"]
Oct 30 04:08:12.259877 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259861    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy"
Oct 30 04:08:12.259877 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259877    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259909    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ec98d1d5-a261-472c-b639-5ce3002206c9" containerName="revoke-keys"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259915    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec98d1d5-a261-472c-b639-5ce3002206c9" containerName="revoke-keys"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259923    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259929    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3bd079-fc0a-4671-b826-b2369583768d" containerName="istio-proxy"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259936    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259941    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy"
Oct 30 04:08:12.259954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259954    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy"
Oct 30 04:08:12.260177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.259959    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy"
Oct 30 04:08:12.260177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.260042    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec98d1d5-a261-472c-b639-5ce3002206c9" containerName="revoke-keys"
Oct 30 04:08:12.260177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.260052    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy"
Oct 30 04:08:12.260177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.260060    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa525320-3c49-4e12-badd-234109092779" containerName="istio-proxy"
Oct 30 04:08:12.260177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.260069    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26db8af-a6b2-412d-87a6-e733209c41f2" containerName="istio-proxy"
Oct 30 04:08:12.263737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.263714    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.280042 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.280017    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:08:12.290542 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.290013    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:08:12.294501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.294474    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"]
Oct 30 04:08:12.348872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.348840    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:12.349047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.348844    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:12.360656 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.360626    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.360913 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.360888    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.360961 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.360930    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5tc\" (UniqueName: \"kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.462136 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.462092    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.462331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.462282    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.462331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.462306    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5tc\" (UniqueName: \"kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.462838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.462817    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.464889 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.464866    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.479008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.478982    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5tc\" (UniqueName: \"kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc\") pod \"maas-api-revoke-sub-311aefa5-s5n94\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.578414 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.578385    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:12.881373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:12.881342    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"]
Oct 30 04:08:12.883229 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:08:12.883202    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7503d4_3753_4199_a664_94a6f6739421.slice/crio-be73e30a1e71c80d20be8865319cba9d6949ab65cfc0bc0d779e6bd12c035d69 WatchSource:0}: Error finding container be73e30a1e71c80d20be8865319cba9d6949ab65cfc0bc0d779e6bd12c035d69: Status 404 returned error can't find the container with id be73e30a1e71c80d20be8865319cba9d6949ab65cfc0bc0d779e6bd12c035d69
Oct 30 04:08:13.345349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.345305    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94" event={"ID":"2e7503d4-3753-4199-a664-94a6f6739421","Type":"ContainerStarted","Data":"8d10a3919fb311f600bba98b74cc218d5d2c23fbba8352c5435d8b762013a1ff"}
Oct 30 04:08:13.345733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.345404    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94" event={"ID":"2e7503d4-3753-4199-a664-94a6f6739421","Type":"ContainerStarted","Data":"be73e30a1e71c80d20be8865319cba9d6949ab65cfc0bc0d779e6bd12c035d69"}
Oct 30 04:08:13.380976 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.380911    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94" podStartSLOduration=1.380893995 podStartE2EDuration="1.380893995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:08:13.378552807 +0000 UTC m=+3921.807442237" watchObservedRunningTime="2026-10-30 04:08:13.380893995 +0000 UTC m=+3921.809783416"
Oct 30 04:08:13.463218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.457512    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"]
Oct 30 04:08:13.463218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.457806    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy" containerID="cri-o://55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9" gracePeriod=30
Oct 30 04:08:13.475044 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.475010    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"]
Oct 30 04:08:13.514143 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:13.514112    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"]
Oct 30 04:08:14.353732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.353698    4877 generic.go:356] "Generic (PLEG): container finished" podID="2e7503d4-3753-4199-a664-94a6f6739421" containerID="8d10a3919fb311f600bba98b74cc218d5d2c23fbba8352c5435d8b762013a1ff" exitCode=0
Oct 30 04:08:14.354210 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.353775    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94" event={"ID":"2e7503d4-3753-4199-a664-94a6f6739421","Type":"ContainerDied","Data":"8d10a3919fb311f600bba98b74cc218d5d2c23fbba8352c5435d8b762013a1ff"}
Oct 30 04:08:14.440146 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.440110    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"]
Oct 30 04:08:14.444488 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.444465    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.464548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.464519    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"]
Oct 30 04:08:14.590869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.590827    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.591083 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.590941    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.591083 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.591012    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxhn\" (UniqueName: \"kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.692131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.692027    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.692131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.692095    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.692131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.692125    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxhn\" (UniqueName: \"kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.692912 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.692885    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.695220 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.695189    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.708980 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.708948    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxhn\" (UniqueName: \"kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn\") pod \"maas-api-revoke-sub-61f3adcc-x46g8\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:14.758088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:14.758061    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:15.077048 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.077017    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"]
Oct 30 04:08:15.078951 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:08:15.078923    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230ebb35_03a1_4859_9ae1_595e063b8faa.slice/crio-70d718022ad4af5b9a043ebb52d9e670386d91ddc60f6cf8ede797c2cd43c781 WatchSource:0}: Error finding container 70d718022ad4af5b9a043ebb52d9e670386d91ddc60f6cf8ede797c2cd43c781: Status 404 returned error can't find the container with id 70d718022ad4af5b9a043ebb52d9e670386d91ddc60f6cf8ede797c2cd43c781
Oct 30 04:08:15.362678 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.362599    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8" event={"ID":"230ebb35-03a1-4859-9ae1-595e063b8faa","Type":"ContainerStarted","Data":"70d718022ad4af5b9a043ebb52d9e670386d91ddc60f6cf8ede797c2cd43c781"}
Oct 30 04:08:15.364459 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.364406    4877 generic.go:356] "Generic (PLEG): container finished" podID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerID="4f497894e4b49cec3818712de391b3ef576c207b4ec19efb6e893673dbec20a8" exitCode=0
Oct 30 04:08:15.364589 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.364458    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" event={"ID":"5f9908a7-9617-4e7f-98a3-7a83b0ddb569","Type":"ContainerDied","Data":"4f497894e4b49cec3818712de391b3ef576c207b4ec19efb6e893673dbec20a8"}
Oct 30 04:08:15.548324 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.548293    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:15.566561 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.566479    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 04:08:15.573970 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.573950    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 04:08:15.605055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.605022    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc\" (UniqueName: \"kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc\") pod \"2e7503d4-3753-4199-a664-94a6f6739421\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") "
Oct 30 04:08:15.605235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.605147    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca\") pod \"2e7503d4-3753-4199-a664-94a6f6739421\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") "
Oct 30 04:08:15.605235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.605214    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca\") pod \"2e7503d4-3753-4199-a664-94a6f6739421\" (UID: \"2e7503d4-3753-4199-a664-94a6f6739421\") "
Oct 30 04:08:15.605548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.605517    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca" pod "2e7503d4-3753-4199-a664-94a6f6739421" (UID: "2e7503d4-3753-4199-a664-94a6f6739421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:15.605701 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.605684    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:15.607526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.607498    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca" pod "2e7503d4-3753-4199-a664-94a6f6739421" (UID: "2e7503d4-3753-4199-a664-94a6f6739421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:15.607647 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.607546    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc" pod "2e7503d4-3753-4199-a664-94a6f6739421" (UID: "2e7503d4-3753-4199-a664-94a6f6739421"). InnerVolumeSpecName "kube-api-access-8r5tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:15.706707 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.706629    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume\") pod \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") "
Oct 30 04:08:15.706881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.706714    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume\") pod \"796823a0-9aee-4c59-be1c-e402d76b6cfe\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") "
Oct 30 04:08:15.706881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.706793    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2\" (UniqueName: \"kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2\") pod \"796823a0-9aee-4c59-be1c-e402d76b6cfe\" (UID: \"796823a0-9aee-4c59-be1c-e402d76b6cfe\") "
Oct 30 04:08:15.706881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.706822    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m\" (UniqueName: \"kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m\") pod \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\" (UID: \"5f9908a7-9617-4e7f-98a3-7a83b0ddb569\") "
Oct 30 04:08:15.707700 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.707560    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume" pod "796823a0-9aee-4c59-be1c-e402d76b6cfe" (UID: "796823a0-9aee-4c59-be1c-e402d76b6cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:15.707872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.707759    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume" pod "5f9908a7-9617-4e7f-98a3-7a83b0ddb569" (UID: "5f9908a7-9617-4e7f-98a3-7a83b0ddb569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:15.707975 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.707719    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5tc\" (UniqueName: \"kubernetes.io/projected/2e7503d4-3753-4199-a664-94a6f6739421-kube-api-access-8r5tc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:15.708096 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.708080    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7503d4-3753-4199-a664-94a6f6739421-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:15.710370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.710324    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2" pod "796823a0-9aee-4c59-be1c-e402d76b6cfe" (UID: "796823a0-9aee-4c59-be1c-e402d76b6cfe"). InnerVolumeSpecName "kube-api-access-5ktc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:15.710370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.710357    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m" pod "5f9908a7-9617-4e7f-98a3-7a83b0ddb569" (UID: "5f9908a7-9617-4e7f-98a3-7a83b0ddb569"). InnerVolumeSpecName "kube-api-access-dbm8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:15.809831 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.809793    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktc2\" (UniqueName: \"kubernetes.io/projected/796823a0-9aee-4c59-be1c-e402d76b6cfe-kube-api-access-5ktc2\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:15.809831 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.809832    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbm8m\" (UniqueName: \"kubernetes.io/projected/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-kube-api-access-dbm8m\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:15.810162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.809851    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9908a7-9617-4e7f-98a3-7a83b0ddb569-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:15.810162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:15.809865    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796823a0-9aee-4c59-be1c-e402d76b6cfe-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:16.229317 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:16.229281    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:08:16.372761 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.372681    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"
Oct 30 04:08:16.372761 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.372681    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94" event={"ID":"2e7503d4-3753-4199-a664-94a6f6739421","Type":"ContainerDied","Data":"be73e30a1e71c80d20be8865319cba9d6949ab65cfc0bc0d779e6bd12c035d69"}
Oct 30 04:08:16.373292 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.372775    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be73e30a1e71c80d20be8865319cba9d6949ab65cfc0bc0d779e6bd12c035d69"
Oct 30 04:08:16.374395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.374369    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"
Oct 30 04:08:16.374395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.374381    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx" event={"ID":"5f9908a7-9617-4e7f-98a3-7a83b0ddb569","Type":"ContainerDied","Data":"f0f59bb413d8393ecbc4fe5f4dd696ee9e5c6027aaf0d918603b83ddc4922159"}
Oct 30 04:08:16.374614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.374453    4877 scope.go:122] "RemoveContainer" containerID="4f497894e4b49cec3818712de391b3ef576c207b4ec19efb6e893673dbec20a8"
Oct 30 04:08:16.376097 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.375939    4877 generic.go:356] "Generic (PLEG): container finished" podID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerID="6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613" exitCode=0
Oct 30 04:08:16.376148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.376105    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" event={"ID":"796823a0-9aee-4c59-be1c-e402d76b6cfe","Type":"ContainerDied","Data":"6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613"}
Oct 30 04:08:16.376194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.376143    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks" event={"ID":"796823a0-9aee-4c59-be1c-e402d76b6cfe","Type":"ContainerDied","Data":"21030dfcfb241219f656001947385ce662e78154f5b50d6f434fc0c8ab7b32aa"}
Oct 30 04:08:16.376238 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.376200    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"
Oct 30 04:08:16.378992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.378965    4877 generic.go:356] "Generic (PLEG): container finished" podID="230ebb35-03a1-4859-9ae1-595e063b8faa" containerID="6dd134be51dbc5ab46866fa856a6e5498e61a47e751381593b46db4b812bb2c3" exitCode=0
Oct 30 04:08:16.379167 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.379103    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8" event={"ID":"230ebb35-03a1-4859-9ae1-595e063b8faa","Type":"ContainerDied","Data":"6dd134be51dbc5ab46866fa856a6e5498e61a47e751381593b46db4b812bb2c3"}
Oct 30 04:08:16.393142 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.393122    4877 scope.go:122] "RemoveContainer" containerID="6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613"
Oct 30 04:08:16.406595 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.406573    4877 scope.go:122] "RemoveContainer" containerID="6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613"
Oct 30 04:08:16.406970 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:16.406949    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613\": container with ID starting with 6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613 not found: ID does not exist" containerID="6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613"
Oct 30 04:08:16.407078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.406976    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613"} err="failed to get container status \"6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613\": rpc error: code = NotFound desc = could not find container \"6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613\": container with ID starting with 6ced85ba2f8191514d69e5a0c5fd32333473ea4f8b0e300a3a76b7059a28e613 not found: ID does not exist"
Oct 30 04:08:16.444897 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.444828    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main" containerID="cri-o://a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2" gracePeriod=45
Oct 30 04:08:16.447573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.447491    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main" containerID="cri-o://b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c" gracePeriod=45
Oct 30 04:08:16.456466 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.456438    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"]
Oct 30 04:08:16.456917 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.456862    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main" containerID="cri-o://141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67" gracePeriod=45
Oct 30 04:08:16.460784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.460761    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7d99a4-cbf456bd-6fcrx"]
Oct 30 04:08:16.514005 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.513975    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"]
Oct 30 04:08:16.518064 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.518038    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-7d99a4-788544f94c-rxtks"]
Oct 30 04:08:16.843855 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.843826    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 04:08:16.846130 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.846108    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 04:08:16.849498 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.849480    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926130    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir\") pod \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926220    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs\" (UniqueName: \"kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs\") pod \"08857070-3f09-450e-affb-e7725c6436db\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926270    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home\") pod \"08857070-3f09-450e-affb-e7725c6436db\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926313    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm\") pod \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926360    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir\") pod \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926392    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm\") pod \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926449    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache\") pod \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926505    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs\" (UniqueName: \"kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs\") pod \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926521    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir" pod "492247f6-5d6b-4060-b7db-a5cec11a4ea5" (UID: "492247f6-5d6b-4060-b7db-a5cec11a4ea5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926556    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l\" (UniqueName: \"kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l\") pod \"08857070-3f09-450e-affb-e7725c6436db\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926605    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache\") pod \"08857070-3f09-450e-affb-e7725c6436db\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926666    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache\") pod \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926711    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74\" (UniqueName: \"kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74\") pod \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926805    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home\") pod \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\" (UID: \"492247f6-5d6b-4060-b7db-a5cec11a4ea5\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926843    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm\") pod \"08857070-3f09-450e-affb-e7725c6436db\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926903    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs\") pod \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926948    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd\" (UniqueName: \"kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd\") pod \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926998    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir\") pod \"08857070-3f09-450e-affb-e7725c6436db\" (UID: \"08857070-3f09-450e-affb-e7725c6436db\") "
Oct 30 04:08:16.927524 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.927030    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home\") pod \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\" (UID: \"6849ee53-403b-499c-baa7-a7ccb92b5ff1\") "
Oct 30 04:08:16.928738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.927659    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.926803    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir" pod "6849ee53-403b-499c-baa7-a7ccb92b5ff1" (UID: "6849ee53-403b-499c-baa7-a7ccb92b5ff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.929255    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home" pod "492247f6-5d6b-4060-b7db-a5cec11a4ea5" (UID: "492247f6-5d6b-4060-b7db-a5cec11a4ea5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.927372    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home" pod "6849ee53-403b-499c-baa7-a7ccb92b5ff1" (UID: "6849ee53-403b-499c-baa7-a7ccb92b5ff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.927959    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home" pod "08857070-3f09-450e-affb-e7725c6436db" (UID: "08857070-3f09-450e-affb-e7725c6436db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.929026    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache" pod "08857070-3f09-450e-affb-e7725c6436db" (UID: "08857070-3f09-450e-affb-e7725c6436db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.929230    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache" pod "6849ee53-403b-499c-baa7-a7ccb92b5ff1" (UID: "6849ee53-403b-499c-baa7-a7ccb92b5ff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.929338    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache" pod "492247f6-5d6b-4060-b7db-a5cec11a4ea5" (UID: "492247f6-5d6b-4060-b7db-a5cec11a4ea5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.929929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.929560    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir" pod "08857070-3f09-450e-affb-e7725c6436db" (UID: "08857070-3f09-450e-affb-e7725c6436db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.932238 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.931875    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm" pod "492247f6-5d6b-4060-b7db-a5cec11a4ea5" (UID: "492247f6-5d6b-4060-b7db-a5cec11a4ea5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.932781 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.932730    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs" pod "08857070-3f09-450e-affb-e7725c6436db" (UID: "08857070-3f09-450e-affb-e7725c6436db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:16.936735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.933918    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs" pod "492247f6-5d6b-4060-b7db-a5cec11a4ea5" (UID: "492247f6-5d6b-4060-b7db-a5cec11a4ea5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:16.936735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.935703    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74" pod "492247f6-5d6b-4060-b7db-a5cec11a4ea5" (UID: "492247f6-5d6b-4060-b7db-a5cec11a4ea5"). InnerVolumeSpecName "kube-api-access-m4b74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:16.936735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.936619    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd" pod "6849ee53-403b-499c-baa7-a7ccb92b5ff1" (UID: "6849ee53-403b-499c-baa7-a7ccb92b5ff1"). InnerVolumeSpecName "kube-api-access-ltpqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:16.936735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.936628    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm" pod "08857070-3f09-450e-affb-e7725c6436db" (UID: "08857070-3f09-450e-affb-e7725c6436db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:16.936735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.936654    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l" pod "08857070-3f09-450e-affb-e7725c6436db" (UID: "08857070-3f09-450e-affb-e7725c6436db"). InnerVolumeSpecName "kube-api-access-w2r7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:16.936735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.936695    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs" pod "6849ee53-403b-499c-baa7-a7ccb92b5ff1" (UID: "6849ee53-403b-499c-baa7-a7ccb92b5ff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:16.940790 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:16.940693    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm" pod "6849ee53-403b-499c-baa7-a7ccb92b5ff1" (UID: "6849ee53-403b-499c-baa7-a7ccb92b5ff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028831    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028867    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028881    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028895    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpqd\" (UniqueName: \"kubernetes.io/projected/6849ee53-403b-499c-baa7-a7ccb92b5ff1-kube-api-access-ltpqd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028908    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028920    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028931    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08857070-3f09-450e-affb-e7725c6436db-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028942    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028953    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028963    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028980    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.028990    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492247f6-5d6b-4060-b7db-a5cec11a4ea5-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.029002    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492247f6-5d6b-4060-b7db-a5cec11a4ea5-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.029015    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2r7l\" (UniqueName: \"kubernetes.io/projected/08857070-3f09-450e-affb-e7725c6436db-kube-api-access-w2r7l\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.029029    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08857070-3f09-450e-affb-e7725c6436db-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.029040    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6849ee53-403b-499c-baa7-a7ccb92b5ff1-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.029175 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.029048    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4b74\" (UniqueName: \"kubernetes.io/projected/492247f6-5d6b-4060-b7db-a5cec11a4ea5-kube-api-access-m4b74\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.392194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.392158    4877 generic.go:356] "Generic (PLEG): container finished" podID="08857070-3f09-450e-affb-e7725c6436db" containerID="141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67" exitCode=0
Oct 30 04:08:17.392714 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.392205    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"
Oct 30 04:08:17.392714 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.392237    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" event={"ID":"08857070-3f09-450e-affb-e7725c6436db","Type":"ContainerDied","Data":"141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67"}
Oct 30 04:08:17.392714 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.392283    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb" event={"ID":"08857070-3f09-450e-affb-e7725c6436db","Type":"ContainerDied","Data":"759bc345f632ae93ac285c8821866900cffbaaa077ca1e334e1b625c6ee996f5"}
Oct 30 04:08:17.392714 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.392301    4877 scope.go:122] "RemoveContainer" containerID="141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67"
Oct 30 04:08:17.393829 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.393805    4877 generic.go:356] "Generic (PLEG): container finished" podID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerID="b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c" exitCode=0
Oct 30 04:08:17.393984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.393967    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" event={"ID":"6849ee53-403b-499c-baa7-a7ccb92b5ff1","Type":"ContainerDied","Data":"b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c"}
Oct 30 04:08:17.394064 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.394001    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85" event={"ID":"6849ee53-403b-499c-baa7-a7ccb92b5ff1","Type":"ContainerDied","Data":"14597affd6e8cbaa368ebcf822d76ea9d203a28b2a77cdf72ffbb25b14aebee5"}
Oct 30 04:08:17.394126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.394082    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"
Oct 30 04:08:17.397197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.397170    4877 generic.go:356] "Generic (PLEG): container finished" podID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerID="a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2" exitCode=0
Oct 30 04:08:17.397294 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.397213    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"
Oct 30 04:08:17.397294 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.397255    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" event={"ID":"492247f6-5d6b-4060-b7db-a5cec11a4ea5","Type":"ContainerDied","Data":"a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2"}
Oct 30 04:08:17.397294 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.397287    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg" event={"ID":"492247f6-5d6b-4060-b7db-a5cec11a4ea5","Type":"ContainerDied","Data":"f16e6eb768e8766ba4e85bd391d6f2d944e66e1657c210c2744d7525768ca253"}
Oct 30 04:08:17.410403 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.410384    4877 scope.go:122] "RemoveContainer" containerID="141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67"
Oct 30 04:08:17.410738 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:17.410711    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67\": container with ID starting with 141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67 not found: ID does not exist" containerID="141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67"
Oct 30 04:08:17.410812 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.410750    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67"} err="failed to get container status \"141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67\": rpc error: code = NotFound desc = could not find container \"141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67\": container with ID starting with 141e4df8f8f4e0e619c0d8ec3454d4c49e758288720303f6c225e9ce74a18f67 not found: ID does not exist"
Oct 30 04:08:17.410812 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.410773    4877 scope.go:122] "RemoveContainer" containerID="b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c"
Oct 30 04:08:17.422932 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.422909    4877 scope.go:122] "RemoveContainer" containerID="b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c"
Oct 30 04:08:17.423270 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:17.423249    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c\": container with ID starting with b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c not found: ID does not exist" containerID="b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c"
Oct 30 04:08:17.423364 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.423275    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c"} err="failed to get container status \"b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c\": rpc error: code = NotFound desc = could not find container \"b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c\": container with ID starting with b01c18d0eefe9fe066754bd4e6aad91a8310dda105db46644d264962a272598c not found: ID does not exist"
Oct 30 04:08:17.423364 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.423291    4877 scope.go:122] "RemoveContainer" containerID="a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2"
Oct 30 04:08:17.434701 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.434686    4877 scope.go:122] "RemoveContainer" containerID="a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2"
Oct 30 04:08:17.434960 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:17.434939    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2\": container with ID starting with a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2 not found: ID does not exist" containerID="a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2"
Oct 30 04:08:17.435019 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.434970    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2"} err="failed to get container status \"a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2\": rpc error: code = NotFound desc = could not find container \"a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2\": container with ID starting with a845688e7156047f2255e2a7cdb0465b8c71ca4a183820d84dfd2c63394261e2 not found: ID does not exist"
Oct 30 04:08:17.463516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.463289    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"]
Oct 30 04:08:17.475001 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.474971    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/simulator-w3-3c874f-kserve-7f85d45f7-d5ktb"]
Oct 30 04:08:17.503515 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.503486    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"]
Oct 30 04:08:17.505463 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.505439    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/distinct-w3-3c874f-kserve-5f7f8dfdd7-clqkg"]
Oct 30 04:08:17.528208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.528188    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:17.533961 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.533936    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"]
Oct 30 04:08:17.539286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.539266    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3c874f/premium-w3-3c874f-kserve-5bb78ffb-svs85"]
Oct 30 04:08:17.636951 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.636916    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn\" (UniqueName: \"kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn\") pod \"230ebb35-03a1-4859-9ae1-595e063b8faa\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") "
Oct 30 04:08:17.637127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.636960    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca\") pod \"230ebb35-03a1-4859-9ae1-595e063b8faa\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") "
Oct 30 04:08:17.637127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.637116    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca\") pod \"230ebb35-03a1-4859-9ae1-595e063b8faa\" (UID: \"230ebb35-03a1-4859-9ae1-595e063b8faa\") "
Oct 30 04:08:17.637532 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.637501    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca" pod "230ebb35-03a1-4859-9ae1-595e063b8faa" (UID: "230ebb35-03a1-4859-9ae1-595e063b8faa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:17.639177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.639156    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn" pod "230ebb35-03a1-4859-9ae1-595e063b8faa" (UID: "230ebb35-03a1-4859-9ae1-595e063b8faa"). InnerVolumeSpecName "kube-api-access-qdxhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:17.639261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.639197    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca" pod "230ebb35-03a1-4859-9ae1-595e063b8faa" (UID: "230ebb35-03a1-4859-9ae1-595e063b8faa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:17.698757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.698678    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08857070-3f09-450e-affb-e7725c6436db" path="/var/lib/kubelet/pods/08857070-3f09-450e-affb-e7725c6436db/volumes"
Oct 30 04:08:17.699110 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.699094    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" path="/var/lib/kubelet/pods/492247f6-5d6b-4060-b7db-a5cec11a4ea5/volumes"
Oct 30 04:08:17.699494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.699478    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" path="/var/lib/kubelet/pods/5f9908a7-9617-4e7f-98a3-7a83b0ddb569/volumes"
Oct 30 04:08:17.700048 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.700027    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" path="/var/lib/kubelet/pods/6849ee53-403b-499c-baa7-a7ccb92b5ff1/volumes"
Oct 30 04:08:17.700546 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.700528    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" path="/var/lib/kubelet/pods/796823a0-9aee-4c59-be1c-e402d76b6cfe/volumes"
Oct 30 04:08:17.741327 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.741273    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxhn\" (UniqueName: \"kubernetes.io/projected/230ebb35-03a1-4859-9ae1-595e063b8faa-kube-api-access-qdxhn\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.741327 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.741313    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:17.741327 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:17.741330    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230ebb35-03a1-4859-9ae1-595e063b8faa-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.409282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.409244    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8" event={"ID":"230ebb35-03a1-4859-9ae1-595e063b8faa","Type":"ContainerDied","Data":"70d718022ad4af5b9a043ebb52d9e670386d91ddc60f6cf8ede797c2cd43c781"}
Oct 30 04:08:18.409736 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.409310    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d718022ad4af5b9a043ebb52d9e670386d91ddc60f6cf8ede797c2cd43c781"
Oct 30 04:08:18.409736 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.409256    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"
Oct 30 04:08:18.475521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.475479    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerName="payload-processing" containerID="cri-o://36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9" gracePeriod=25
Oct 30 04:08:18.515099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.515065    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerName="payload-pre-processing" containerID="cri-o://24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4" gracePeriod=25
Oct 30 04:08:18.741452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.741415    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 04:08:18.852656 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.852622    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.852827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.852691    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.852827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.852725    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.852827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.852761    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.853127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.853102    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.853251 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.853236    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.853361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.853347    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.853527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.853502    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.853737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.853584    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7\") pod \"cafd624a-1b4d-4710-a301-d6dff69bb92c\" (UID: \"cafd624a-1b4d-4710-a301-d6dff69bb92c\") "
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.855277    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.855406    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.855480    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.855553    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.855685    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.855997    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:18.860452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.857526    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:18.864229 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.863953    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "kube-api-access-fjzq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:18.880707 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.878509    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo" pod "cafd624a-1b4d-4710-a301-d6dff69bb92c" (UID: "cafd624a-1b4d-4710-a301-d6dff69bb92c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:08:18.955499 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955467    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955499 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955495    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955507    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955518    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955526    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955535    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955543    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafd624a-1b4d-4710-a301-d6dff69bb92c-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955551    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjzq7\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-kube-api-access-fjzq7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:18.955698 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:18.955559    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafd624a-1b4d-4710-a301-d6dff69bb92c-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:19.420715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.420675    4877 generic.go:356] "Generic (PLEG): container finished" podID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerID="55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9" exitCode=0
Oct 30 04:08:19.421235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.420750    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"
Oct 30 04:08:19.421235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.420753    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" event={"ID":"cafd624a-1b4d-4710-a301-d6dff69bb92c","Type":"ContainerDied","Data":"55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9"}
Oct 30 04:08:19.421235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.420858    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh" event={"ID":"cafd624a-1b4d-4710-a301-d6dff69bb92c","Type":"ContainerDied","Data":"476bc90310706d2948fe36910bea0fcf6be6bc59b3a4b4cced9032e2f50eddfd"}
Oct 30 04:08:19.421235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.420899    4877 scope.go:122] "RemoveContainer" containerID="55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9"
Oct 30 04:08:19.437342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.436558    4877 scope.go:122] "RemoveContainer" containerID="04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"
Oct 30 04:08:19.447807 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.447785    4877 scope.go:122] "RemoveContainer" containerID="55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9"
Oct 30 04:08:19.448075 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:19.448051    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9\": container with ID starting with 55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9 not found: ID does not exist" containerID="55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9"
Oct 30 04:08:19.448134 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.448082    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9"} err="failed to get container status \"55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9\": rpc error: code = NotFound desc = could not find container \"55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9\": container with ID starting with 55090a5cabd943b4caf6e193da1ed9bc91b4ce5227aa0f21d114d4bae55f3ec9 not found: ID does not exist"
Oct 30 04:08:19.448134 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.448098    4877 scope.go:122] "RemoveContainer" containerID="04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"
Oct 30 04:08:19.448368 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:19.448349    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469\": container with ID starting with 04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469 not found: ID does not exist" containerID="04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"
Oct 30 04:08:19.448437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.448375    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469"} err="failed to get container status \"04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469\": rpc error: code = NotFound desc = could not find container \"04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469\": container with ID starting with 04f3376ac4185d8757eb424ef48e28ed546a87cf5a818c9bee4492de677ff469 not found: ID does not exist"
Oct 30 04:08:19.480734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.480705    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"]
Oct 30 04:08:19.489061 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.489035    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-852107-openshift-default-dbf795c59-5hznh"]
Oct 30 04:08:19.699356 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:19.699277    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" path="/var/lib/kubelet/pods/cafd624a-1b4d-4710-a301-d6dff69bb92c/volumes"
Oct 30 04:08:24.443308 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:24.443270    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:25.324689 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.324654    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"]
Oct 30 04:08:25.347204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.344817    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"]
Oct 30 04:08:25.453478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.453391    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:25.688482 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.688333    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"]
Oct 30 04:08:25.688986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.688960    4877 cpu_manager.go:414] "containerMap: removing container" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.688988    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.688999    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2e7503d4-3753-4199-a664-94a6f6739421" containerName="revoke-keys"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689004    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7503d4-3753-4199-a664-94a6f6739421" containerName="revoke-keys"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689016    4877 cpu_manager.go:414] "containerMap: removing container" podUID="230ebb35-03a1-4859-9ae1-595e063b8faa" containerName="revoke-keys"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689021    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230ebb35-03a1-4859-9ae1-595e063b8faa" containerName="revoke-keys"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689043    4877 cpu_manager.go:414] "containerMap: removing container" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerName="payload-pre-processing"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689049    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerName="payload-pre-processing"
Oct 30 04:08:25.689056 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689057    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689062    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689070    4877 cpu_manager.go:414] "containerMap: removing container" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689074    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689081    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerName="payload-processing"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689085    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerName="payload-processing"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689094    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689098    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689109    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689115    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689195    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="796823a0-9aee-4c59-be1c-e402d76b6cfe" containerName="payload-pre-processing"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689205    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689211    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="492247f6-5d6b-4060-b7db-a5cec11a4ea5" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689216    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9908a7-9617-4e7f-98a3-7a83b0ddb569" containerName="payload-processing"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689221    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafd624a-1b4d-4710-a301-d6dff69bb92c" containerName="istio-proxy"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689229    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6849ee53-403b-499c-baa7-a7ccb92b5ff1" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689236    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="08857070-3f09-450e-affb-e7725c6436db" containerName="main"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689242    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7503d4-3753-4199-a664-94a6f6739421" containerName="revoke-keys"
Oct 30 04:08:25.689330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.689249    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="230ebb35-03a1-4859-9ae1-595e063b8faa" containerName="revoke-keys"
Oct 30 04:08:25.694479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.694452    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.707960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.707935    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:08:25.708149 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.708126    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:08:25.712571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.712546    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"]
Oct 30 04:08:25.743991 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.743958    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.744144 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.744005    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbms\" (UniqueName: \"kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.744243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.744222    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.845602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.845562    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.845803 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.845654    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.845803 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.845677    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbms\" (UniqueName: \"kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.846400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.846379    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.848049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.848024    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:25.862716 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:25.862691    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbms\" (UniqueName: \"kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms\") pod \"maas-api-revoke-sub-708a6f4a-r4z8c\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:26.010838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:26.010752    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:26.317191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:26.317164    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"]
Oct 30 04:08:26.318567 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:08:26.318533    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8438eaf6_ec90_433e_9097_383466506fed.slice/crio-97caf77c9e024d4c5c98795eda0f7182bd2e729f98dcd51d2549a5456ed20fcd WatchSource:0}: Error finding container 97caf77c9e024d4c5c98795eda0f7182bd2e729f98dcd51d2549a5456ed20fcd: Status 404 returned error can't find the container with id 97caf77c9e024d4c5c98795eda0f7182bd2e729f98dcd51d2549a5456ed20fcd
Oct 30 04:08:26.475969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:26.475933    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c" event={"ID":"8438eaf6-ec90-433e-9097-383466506fed","Type":"ContainerStarted","Data":"97caf77c9e024d4c5c98795eda0f7182bd2e729f98dcd51d2549a5456ed20fcd"}
Oct 30 04:08:27.485819 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:27.485728    4877 generic.go:356] "Generic (PLEG): container finished" podID="8438eaf6-ec90-433e-9097-383466506fed" containerID="3e5ab2f219017bdf0a3b1d195e5231f51b48ce3b9fdedabe028d3291cf88eb4c" exitCode=0
Oct 30 04:08:27.485819 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:27.485797    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c" event={"ID":"8438eaf6-ec90-433e-9097-383466506fed","Type":"ContainerDied","Data":"3e5ab2f219017bdf0a3b1d195e5231f51b48ce3b9fdedabe028d3291cf88eb4c"}
Oct 30 04:08:28.644492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.644469    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:28.675828 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.675795    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca\") pod \"8438eaf6-ec90-433e-9097-383466506fed\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") "
Oct 30 04:08:28.676007 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.675865    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca\") pod \"8438eaf6-ec90-433e-9097-383466506fed\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") "
Oct 30 04:08:28.676007 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.675931    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms\" (UniqueName: \"kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms\") pod \"8438eaf6-ec90-433e-9097-383466506fed\" (UID: \"8438eaf6-ec90-433e-9097-383466506fed\") "
Oct 30 04:08:28.676170 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.676143    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca" pod "8438eaf6-ec90-433e-9097-383466506fed" (UID: "8438eaf6-ec90-433e-9097-383466506fed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:28.676480 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.676409    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8438eaf6-ec90-433e-9097-383466506fed-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:28.678280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.678252    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms" pod "8438eaf6-ec90-433e-9097-383466506fed" (UID: "8438eaf6-ec90-433e-9097-383466506fed"). InnerVolumeSpecName "kube-api-access-sdbms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:28.678641 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.678618    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca" pod "8438eaf6-ec90-433e-9097-383466506fed" (UID: "8438eaf6-ec90-433e-9097-383466506fed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:08:28.777893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.777858    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8438eaf6-ec90-433e-9097-383466506fed-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:28.777893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:28.777889    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbms\" (UniqueName: \"kubernetes.io/projected/8438eaf6-ec90-433e-9097-383466506fed-kube-api-access-sdbms\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:29.500902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:29.500865    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"
Oct 30 04:08:29.500902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:29.500894    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c" event={"ID":"8438eaf6-ec90-433e-9097-383466506fed","Type":"ContainerDied","Data":"97caf77c9e024d4c5c98795eda0f7182bd2e729f98dcd51d2549a5456ed20fcd"}
Oct 30 04:08:29.501150 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:29.500984    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97caf77c9e024d4c5c98795eda0f7182bd2e729f98dcd51d2549a5456ed20fcd"
Oct 30 04:08:30.325390 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:30.325334    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerName="payload-processing" containerID="cri-o://b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1" gracePeriod=25
Oct 30 04:08:30.345939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:30.345897    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerName="payload-pre-processing" containerID="cri-o://dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a" gracePeriod=25
Oct 30 04:08:33.436494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:33.436451    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:33.436986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:33.436871    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:34.442373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:34.442324    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:35.452312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:35.452274    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:38.782838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.782812    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 04:08:38.810791 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.810765    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 04:08:38.935789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.935752    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm\" (UniqueName: \"kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm\") pod \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") "
Oct 30 04:08:38.935994 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.935825    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume\") pod \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\" (UID: \"cb72dc63-2e16-4bef-beb9-ae5ce2605774\") "
Oct 30 04:08:38.935994 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.935855    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume\") pod \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") "
Oct 30 04:08:38.935994 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.935891    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs\" (UniqueName: \"kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs\") pod \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\" (UID: \"cfe2da65-df59-4c30-bb1c-57a33bc91abd\") "
Oct 30 04:08:38.936306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.936277    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume" pod "cb72dc63-2e16-4bef-beb9-ae5ce2605774" (UID: "cb72dc63-2e16-4bef-beb9-ae5ce2605774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:38.936393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.936304    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume" pod "cfe2da65-df59-4c30-bb1c-57a33bc91abd" (UID: "cfe2da65-df59-4c30-bb1c-57a33bc91abd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:38.938061 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.938030    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm" pod "cb72dc63-2e16-4bef-beb9-ae5ce2605774" (UID: "cb72dc63-2e16-4bef-beb9-ae5ce2605774"). InnerVolumeSpecName "kube-api-access-c7qsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:38.938165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:38.938072    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs" pod "cfe2da65-df59-4c30-bb1c-57a33bc91abd" (UID: "cfe2da65-df59-4c30-bb1c-57a33bc91abd"). InnerVolumeSpecName "kube-api-access-vcmjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:39.036880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.036847    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmjs\" (UniqueName: \"kubernetes.io/projected/cfe2da65-df59-4c30-bb1c-57a33bc91abd-kube-api-access-vcmjs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:39.036880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.036876    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qsm\" (UniqueName: \"kubernetes.io/projected/cb72dc63-2e16-4bef-beb9-ae5ce2605774-kube-api-access-c7qsm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:39.036880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.036886    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb72dc63-2e16-4bef-beb9-ae5ce2605774-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:39.037145 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.036894    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe2da65-df59-4c30-bb1c-57a33bc91abd-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:39.589012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.588978    4877 generic.go:356] "Generic (PLEG): container finished" podID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerID="36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9" exitCode=0
Oct 30 04:08:39.589202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.589026    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"
Oct 30 04:08:39.589202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.589067    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" event={"ID":"cfe2da65-df59-4c30-bb1c-57a33bc91abd","Type":"ContainerDied","Data":"36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9"}
Oct 30 04:08:39.589202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.589129    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2" event={"ID":"cfe2da65-df59-4c30-bb1c-57a33bc91abd","Type":"ContainerDied","Data":"bb269c92dbf2fb9374377ab54fe7624ce9b96fc7fca06a16c1fc069e5333cdee"}
Oct 30 04:08:39.589202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.589156    4877 scope.go:122] "RemoveContainer" containerID="36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9"
Oct 30 04:08:39.590910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.590882    4877 generic.go:356] "Generic (PLEG): container finished" podID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerID="24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4" exitCode=0
Oct 30 04:08:39.591047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.590933    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"
Oct 30 04:08:39.591047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.590956    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" event={"ID":"cb72dc63-2e16-4bef-beb9-ae5ce2605774","Type":"ContainerDied","Data":"24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4"}
Oct 30 04:08:39.591047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.591000    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq" event={"ID":"cb72dc63-2e16-4bef-beb9-ae5ce2605774","Type":"ContainerDied","Data":"9568d37ca4f1aeb9bb7107464598126ca5e02c2826056b68fbbac9aebf54b0fe"}
Oct 30 04:08:39.627357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.627181    4877 scope.go:122] "RemoveContainer" containerID="36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9"
Oct 30 04:08:39.627514 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:39.627490    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9\": container with ID starting with 36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9 not found: ID does not exist" containerID="36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9"
Oct 30 04:08:39.627567 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.627524    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9"} err="failed to get container status \"36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9\": rpc error: code = NotFound desc = could not find container \"36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9\": container with ID starting with 36fff4b99c1b41503a56d685dad5863af9e436c5772574ad12f831dfc5c700a9 not found: ID does not exist"
Oct 30 04:08:39.627567 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.627542    4877 scope.go:122] "RemoveContainer" containerID="24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4"
Oct 30 04:08:39.637881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.637863    4877 scope.go:122] "RemoveContainer" containerID="24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4"
Oct 30 04:08:39.638180 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:39.638158    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4\": container with ID starting with 24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4 not found: ID does not exist" containerID="24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4"
Oct 30 04:08:39.638253 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.638192    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4"} err="failed to get container status \"24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4\": rpc error: code = NotFound desc = could not find container \"24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4\": container with ID starting with 24a92d180ab3b8efaa1b8c68870effbe86c853a4714d1b76f1972ab659b93fb4 not found: ID does not exist"
Oct 30 04:08:39.645927 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.645906    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"]
Oct 30 04:08:39.655534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.655514    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3c874f-998846cc4-tbtz2"]
Oct 30 04:08:39.683537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.683504    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"]
Oct 30 04:08:39.687039 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.687012    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3c874f-57c4bc9655-pqvqq"]
Oct 30 04:08:39.697632 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.697608    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" path="/var/lib/kubelet/pods/cb72dc63-2e16-4bef-beb9-ae5ce2605774/volumes"
Oct 30 04:08:39.698021 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:39.698009    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" path="/var/lib/kubelet/pods/cfe2da65-df59-4c30-bb1c-57a33bc91abd/volumes"
Oct 30 04:08:43.436942 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:43.436888    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:43.437459 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:43.437068    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:08:46.229056 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:46.229017    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:08:48.399824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:48.399782    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"]
Oct 30 04:08:48.400340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:48.400041    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" containerID="cri-o://f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710" gracePeriod=30
Oct 30 04:08:50.625007 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.624975    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 04:08:50.645139 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.645118    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 04:08:50.686407 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.686362    4877 generic.go:356] "Generic (PLEG): container finished" podID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerID="dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a" exitCode=0
Oct 30 04:08:50.686597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.686450    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"
Oct 30 04:08:50.686597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.686564    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" event={"ID":"ac5a933d-5487-44e3-ac11-855d8bf48846","Type":"ContainerDied","Data":"dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a"}
Oct 30 04:08:50.686734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.686618    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4" event={"ID":"ac5a933d-5487-44e3-ac11-855d8bf48846","Type":"ContainerDied","Data":"d063cf7208e04ba2a3555b3d2b8f32544c3ac04bf310e515e81eaf5227d1b0c0"}
Oct 30 04:08:50.686734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.686651    4877 scope.go:122] "RemoveContainer" containerID="dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a"
Oct 30 04:08:50.688458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.688155    4877 generic.go:356] "Generic (PLEG): container finished" podID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerID="b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1" exitCode=0
Oct 30 04:08:50.688458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.688199    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"
Oct 30 04:08:50.688458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.688228    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" event={"ID":"21fb84a1-a244-4ca4-9b9c-070c2859b235","Type":"ContainerDied","Data":"b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1"}
Oct 30 04:08:50.688458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.688267    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r" event={"ID":"21fb84a1-a244-4ca4-9b9c-070c2859b235","Type":"ContainerDied","Data":"c38686160d72c84f915f4e32e9c5564eeb06e50ddbc79977e532b5ec7fd414cd"}
Oct 30 04:08:50.707705 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.707678    4877 scope.go:122] "RemoveContainer" containerID="dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a"
Oct 30 04:08:50.708095 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:50.708062    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a\": container with ID starting with dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a not found: ID does not exist" containerID="dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a"
Oct 30 04:08:50.708232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.708103    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a"} err="failed to get container status \"dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a\": rpc error: code = NotFound desc = could not find container \"dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a\": container with ID starting with dd5622ddb00ad812892ad0d8e500219320352c79d76b9169e284e5f40e5a160a not found: ID does not exist"
Oct 30 04:08:50.708232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.708122    4877 scope.go:122] "RemoveContainer" containerID="b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1"
Oct 30 04:08:50.728880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.726568    4877 scope.go:122] "RemoveContainer" containerID="b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1"
Oct 30 04:08:50.728880 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:50.727015    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1\": container with ID starting with b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1 not found: ID does not exist" containerID="b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1"
Oct 30 04:08:50.728880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.727061    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1"} err="failed to get container status \"b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1\": rpc error: code = NotFound desc = could not find container \"b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1\": container with ID starting with b0908817485a21a73509bd6277cdde97133a7345a21878823b5751d6495b44e1 not found: ID does not exist"
Oct 30 04:08:50.776964 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.776936    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs\" (UniqueName: \"kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs\") pod \"21fb84a1-a244-4ca4-9b9c-070c2859b235\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") "
Oct 30 04:08:50.777169 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.777150    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume\") pod \"ac5a933d-5487-44e3-ac11-855d8bf48846\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") "
Oct 30 04:08:50.777249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.777192    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm\" (UniqueName: \"kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm\") pod \"ac5a933d-5487-44e3-ac11-855d8bf48846\" (UID: \"ac5a933d-5487-44e3-ac11-855d8bf48846\") "
Oct 30 04:08:50.777249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.777236    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume\") pod \"21fb84a1-a244-4ca4-9b9c-070c2859b235\" (UID: \"21fb84a1-a244-4ca4-9b9c-070c2859b235\") "
Oct 30 04:08:50.777648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.777607    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume" pod "ac5a933d-5487-44e3-ac11-855d8bf48846" (UID: "ac5a933d-5487-44e3-ac11-855d8bf48846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:50.777755 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.777644    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume" pod "21fb84a1-a244-4ca4-9b9c-070c2859b235" (UID: "21fb84a1-a244-4ca4-9b9c-070c2859b235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:50.779124 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.779099    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs" pod "21fb84a1-a244-4ca4-9b9c-070c2859b235" (UID: "21fb84a1-a244-4ca4-9b9c-070c2859b235"). InnerVolumeSpecName "kube-api-access-mrlqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:50.779298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.779276    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm" pod "ac5a933d-5487-44e3-ac11-855d8bf48846" (UID: "ac5a933d-5487-44e3-ac11-855d8bf48846"). InnerVolumeSpecName "kube-api-access-td4mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:50.878472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.878378    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5a933d-5487-44e3-ac11-855d8bf48846-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:50.878472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.878411    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4mm\" (UniqueName: \"kubernetes.io/projected/ac5a933d-5487-44e3-ac11-855d8bf48846-kube-api-access-td4mm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:50.878472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.878442    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb84a1-a244-4ca4-9b9c-070c2859b235-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:50.878472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:50.878458    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrlqs\" (UniqueName: \"kubernetes.io/projected/21fb84a1-a244-4ca4-9b9c-070c2859b235-kube-api-access-mrlqs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:51.033046 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.033017    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"]
Oct 30 04:08:51.036332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.036308    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8d80b1-75767f48bc-ggqd4"]
Oct 30 04:08:51.062358 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.062329    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"]
Oct 30 04:08:51.065508 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.065485    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8d80b1-7db8ddbb44-dbk8r"]
Oct 30 04:08:51.532095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.532064    4877 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:08:51.532242 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.532131    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:08:51.699331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.699247    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" path="/var/lib/kubelet/pods/21fb84a1-a244-4ca4-9b9c-070c2859b235/volumes"
Oct 30 04:08:51.699838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:51.699817    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" path="/var/lib/kubelet/pods/ac5a933d-5487-44e3-ac11-855d8bf48846/volumes"
Oct 30 04:08:53.689890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.689827    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 04:08:53.711407 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711371    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711595 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711455    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711595 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711490    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711595 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711519    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711629    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711671    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711717    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.711770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711725    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:53.711770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711763    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:08:53.712060 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711766    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.712060 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711852    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99\") pod \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\" (UID: \"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5\") "
Oct 30 04:08:53.712060 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.711943    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:53.712225 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.712114    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:53.712399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.712368    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.712399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.712394    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.712562 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.712407    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.712562 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.712436    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.712671 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.712635    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:53.714345 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.714291    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:53.714913 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.714883    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:08:53.715095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.715060    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "kube-api-access-gxr99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:08:53.715206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.715142    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo" pod "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" (UID: "e7dfaff2-04a1-4e93-828f-6ce9fbc495e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:08:53.717233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.717207    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerID="f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710" exitCode=0
Oct 30 04:08:53.717341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.717246    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"
Oct 30 04:08:53.717341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.717251    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" event={"ID":"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5","Type":"ContainerDied","Data":"f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710"}
Oct 30 04:08:53.717341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.717298    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd" event={"ID":"e7dfaff2-04a1-4e93-828f-6ce9fbc495e5","Type":"ContainerDied","Data":"d0cacd9ffb7c8abaf897673e44bd124dd49ad3bf59a4dfe93fb049a1d8d3fbfa"}
Oct 30 04:08:53.717341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.717325    4877 scope.go:122] "RemoveContainer" containerID="f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710"
Oct 30 04:08:53.753679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.753652    4877 scope.go:122] "RemoveContainer" containerID="f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710"
Oct 30 04:08:53.754152 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:08:53.754115    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710\": container with ID starting with f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710 not found: ID does not exist" containerID="f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710"
Oct 30 04:08:53.754239 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.754167    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710"} err="failed to get container status \"f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710\": rpc error: code = NotFound desc = could not find container \"f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710\": container with ID starting with f651fcccf0409fc400cf05e3c98d1133f948388dc09c4eeebe162d11f065d710 not found: ID does not exist"
Oct 30 04:08:53.770263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.770234    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"]
Oct 30 04:08:53.775081 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.775055    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-mrcmd"]
Oct 30 04:08:53.813239 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.813210    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.813239 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.813235    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.813411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.813244    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.813411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.813255    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:53.813411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:53.813264    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxr99\" (UniqueName: \"kubernetes.io/projected/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5-kube-api-access-gxr99\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:08:55.700135 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:55.700098    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" path="/var/lib/kubelet/pods/e7dfaff2-04a1-4e93-828f-6ce9fbc495e5/volumes"
Oct 30 04:08:57.504765 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:08:57.504730    4877 scope.go:122] "RemoveContainer" containerID="f982c3a04c542bc737531d43939e6c7ae269b70e4a7c65e96f6fca4ce6877b62"
Oct 30 04:09:06.598872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.598829    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"]
Oct 30 04:09:06.599369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599326    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerName="payload-pre-processing"
Oct 30 04:09:06.599369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599341    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerName="payload-pre-processing"
Oct 30 04:09:06.599369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599360    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8438eaf6-ec90-433e-9097-383466506fed" containerName="revoke-keys"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599368    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8438eaf6-ec90-433e-9097-383466506fed" containerName="revoke-keys"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599387    4877 cpu_manager.go:414] "containerMap: removing container" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerName="payload-processing"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599392    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerName="payload-processing"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599403    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599408    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599435    4877 cpu_manager.go:414] "containerMap: removing container" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerName="payload-processing"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599440    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerName="payload-processing"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599448    4877 cpu_manager.go:414] "containerMap: removing container" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerName="payload-pre-processing"
Oct 30 04:09:06.599520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599453    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerName="payload-pre-processing"
Oct 30 04:09:06.599810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599530    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb72dc63-2e16-4bef-beb9-ae5ce2605774" containerName="payload-pre-processing"
Oct 30 04:09:06.599810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599537    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8438eaf6-ec90-433e-9097-383466506fed" containerName="revoke-keys"
Oct 30 04:09:06.599810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599544    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="21fb84a1-a244-4ca4-9b9c-070c2859b235" containerName="payload-processing"
Oct 30 04:09:06.599810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599551    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dfaff2-04a1-4e93-828f-6ce9fbc495e5" containerName="istio-proxy"
Oct 30 04:09:06.599810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599559    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe2da65-df59-4c30-bb1c-57a33bc91abd" containerName="payload-processing"
Oct 30 04:09:06.599810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.599565    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5a933d-5487-44e3-ac11-855d8bf48846" containerName="payload-pre-processing"
Oct 30 04:09:06.602467 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.602449    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.624730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.624708    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:09:06.624861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.624711    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:09:06.630279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.630252    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"]
Oct 30 04:09:06.665088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.665051    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.665278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.665183    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.665278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.665211    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvq46\" (UniqueName: \"kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.766623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.766588    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.766823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.766636    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvq46\" (UniqueName: \"kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.766823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.766771    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.767305 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.767282    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.769270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.769247    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.784346 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.784325    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvq46\" (UniqueName: \"kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46\") pod \"maas-api-revoke-sub-e51b37a0-rt6zs\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:06.916774 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:06.916698    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:07.221509 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:07.221482    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"]
Oct 30 04:09:07.223947 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:09:07.223922    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9933ea9c_eebb_45e7_93b4_4b56fc4dfd15.slice/crio-a5146db7a311b3d4e06022c9190ddf02e6da565aa5bb8c07e3656d52b3abf87e WatchSource:0}: Error finding container a5146db7a311b3d4e06022c9190ddf02e6da565aa5bb8c07e3656d52b3abf87e: Status 404 returned error can't find the container with id a5146db7a311b3d4e06022c9190ddf02e6da565aa5bb8c07e3656d52b3abf87e
Oct 30 04:09:07.822224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:07.822185    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs" event={"ID":"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15","Type":"ContainerStarted","Data":"41017cb05f335da0cf1a18b697f80640f40b5dc2945b4f69b978daa4a446d6c6"}
Oct 30 04:09:07.822639 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:07.822246    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs" event={"ID":"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15","Type":"ContainerStarted","Data":"a5146db7a311b3d4e06022c9190ddf02e6da565aa5bb8c07e3656d52b3abf87e"}
Oct 30 04:09:07.859629 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:07.859548    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs" podStartSLOduration=1.859530227 podStartE2EDuration="1.859530227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:07.856952416 +0000 UTC m=+3976.285841835" watchObservedRunningTime="2026-10-30 04:09:07.859530227 +0000 UTC m=+3976.288419645"
Oct 30 04:09:08.831025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:08.830983    4877 generic.go:356] "Generic (PLEG): container finished" podID="9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" containerID="41017cb05f335da0cf1a18b697f80640f40b5dc2945b4f69b978daa4a446d6c6" exitCode=0
Oct 30 04:09:08.831507 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:08.831064    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs" event={"ID":"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15","Type":"ContainerDied","Data":"41017cb05f335da0cf1a18b697f80640f40b5dc2945b4f69b978daa4a446d6c6"}
Oct 30 04:09:09.994604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:09.994576    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:10.101213 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.101129    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46\" (UniqueName: \"kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46\") pod \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") "
Oct 30 04:09:10.101213 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.101183    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca\") pod \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") "
Oct 30 04:09:10.101495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.101256    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca\") pod \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\" (UID: \"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15\") "
Oct 30 04:09:10.101653 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.101627    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca" pod "9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" (UID: "9933ea9c-eebb-45e7-93b4-4b56fc4dfd15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:10.103360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.103325    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca" pod "9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" (UID: "9933ea9c-eebb-45e7-93b4-4b56fc4dfd15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:10.103483 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.103382    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46" pod "9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" (UID: "9933ea9c-eebb-45e7-93b4-4b56fc4dfd15"). InnerVolumeSpecName "kube-api-access-jvq46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:10.203036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.202999    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:09:10.203036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.203037    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvq46\" (UniqueName: \"kubernetes.io/projected/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-kube-api-access-jvq46\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:09:10.203251 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.203052    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:09:10.849540 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.849500    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"
Oct 30 04:09:10.849540 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.849517    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs" event={"ID":"9933ea9c-eebb-45e7-93b4-4b56fc4dfd15","Type":"ContainerDied","Data":"a5146db7a311b3d4e06022c9190ddf02e6da565aa5bb8c07e3656d52b3abf87e"}
Oct 30 04:09:10.849757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:10.849564    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5146db7a311b3d4e06022c9190ddf02e6da565aa5bb8c07e3656d52b3abf87e"
Oct 30 04:09:16.228309 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:09:16.228270    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:09:16.897211 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.897177    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"]
Oct 30 04:09:16.897680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.897665    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" containerName="revoke-keys"
Oct 30 04:09:16.897680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.897679    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" containerName="revoke-keys"
Oct 30 04:09:16.897835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.897784    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" containerName="revoke-keys"
Oct 30 04:09:16.901311 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.901291    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:16.921182 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.921154    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"]
Oct 30 04:09:16.946228 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.946203    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:09:16.946352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.946277    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:09:16.985621 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.985595    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:16.985751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.985631    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhl2x\" (UniqueName: \"kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:16.985751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:16.985654    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.086958 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.086926    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.087092 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.086967    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhl2x\" (UniqueName: \"kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.087092 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.086999    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.087672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.087650    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.089330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.089304    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.104447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.104413    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhl2x\" (UniqueName: \"kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x\") pod \"maas-api-revoke-sub-271dc16c-2qqpd\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.214235 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.214159    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:17.514492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.514464    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"]
Oct 30 04:09:17.516605 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:09:17.516578    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c572573_47ef_4b27_9e13_c1a561d30b9f.slice/crio-1199ee9c40b8de65d03b87850c98cde2dc8103942cf49f609adc1f41e2580128 WatchSource:0}: Error finding container 1199ee9c40b8de65d03b87850c98cde2dc8103942cf49f609adc1f41e2580128: Status 404 returned error can't find the container with id 1199ee9c40b8de65d03b87850c98cde2dc8103942cf49f609adc1f41e2580128
Oct 30 04:09:17.899921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.899888    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd" event={"ID":"1c572573-47ef-4b27-9e13-c1a561d30b9f","Type":"ContainerStarted","Data":"16eff3a2a7e167f96b59c51993505181258cdf7b87d379764079dfc4bd0bf8c8"}
Oct 30 04:09:17.900135 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.899944    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd" event={"ID":"1c572573-47ef-4b27-9e13-c1a561d30b9f","Type":"ContainerStarted","Data":"1199ee9c40b8de65d03b87850c98cde2dc8103942cf49f609adc1f41e2580128"}
Oct 30 04:09:17.934676 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:17.934625    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd" podStartSLOduration=1.934611006 podStartE2EDuration="1.934611006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:17.933317836 +0000 UTC m=+3986.362207256" watchObservedRunningTime="2026-10-30 04:09:17.934611006 +0000 UTC m=+3986.363500424"
Oct 30 04:09:18.910058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:18.910018    4877 generic.go:356] "Generic (PLEG): container finished" podID="1c572573-47ef-4b27-9e13-c1a561d30b9f" containerID="16eff3a2a7e167f96b59c51993505181258cdf7b87d379764079dfc4bd0bf8c8" exitCode=0
Oct 30 04:09:18.910584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:18.910114    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd" event={"ID":"1c572573-47ef-4b27-9e13-c1a561d30b9f","Type":"ContainerDied","Data":"16eff3a2a7e167f96b59c51993505181258cdf7b87d379764079dfc4bd0bf8c8"}
Oct 30 04:09:20.102758 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.102732    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:20.128614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.128588    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x\" (UniqueName: \"kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x\") pod \"1c572573-47ef-4b27-9e13-c1a561d30b9f\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") "
Oct 30 04:09:20.130660 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.130633    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x" pod "1c572573-47ef-4b27-9e13-c1a561d30b9f" (UID: "1c572573-47ef-4b27-9e13-c1a561d30b9f"). InnerVolumeSpecName "kube-api-access-jhl2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:20.229646 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.229556    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca\") pod \"1c572573-47ef-4b27-9e13-c1a561d30b9f\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") "
Oct 30 04:09:20.229646 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.229612    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca\") pod \"1c572573-47ef-4b27-9e13-c1a561d30b9f\" (UID: \"1c572573-47ef-4b27-9e13-c1a561d30b9f\") "
Oct 30 04:09:20.229903 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.229846    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhl2x\" (UniqueName: \"kubernetes.io/projected/1c572573-47ef-4b27-9e13-c1a561d30b9f-kube-api-access-jhl2x\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:09:20.230066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.230037    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca" pod "1c572573-47ef-4b27-9e13-c1a561d30b9f" (UID: "1c572573-47ef-4b27-9e13-c1a561d30b9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:09:20.231733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.231707    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca" pod "1c572573-47ef-4b27-9e13-c1a561d30b9f" (UID: "1c572573-47ef-4b27-9e13-c1a561d30b9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:20.331016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.330977    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:09:20.331016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.331012    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c572573-47ef-4b27-9e13-c1a561d30b9f-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:09:20.924779 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.924743    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd" event={"ID":"1c572573-47ef-4b27-9e13-c1a561d30b9f","Type":"ContainerDied","Data":"1199ee9c40b8de65d03b87850c98cde2dc8103942cf49f609adc1f41e2580128"}
Oct 30 04:09:20.924976 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.924803    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1199ee9c40b8de65d03b87850c98cde2dc8103942cf49f609adc1f41e2580128"
Oct 30 04:09:20.924976 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:20.924751    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"
Oct 30 04:09:42.034860 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:42.034751    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"]
Oct 30 04:09:42.043923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:42.043894    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8383a775-zxtdk"]
Oct 30 04:09:43.697922 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:43.697888    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f409fe59-079b-49a9-992c-9bd76d3db66d" path="/var/lib/kubelet/pods/f409fe59-079b-49a9-992c-9bd76d3db66d/volumes"
Oct 30 04:09:46.025977 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:46.025940    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"]
Oct 30 04:09:46.029232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:46.029211    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971206c4-lhgb4"]
Oct 30 04:09:46.230887 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:09:46.230846    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:09:47.701987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:47.701953    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3" path="/var/lib/kubelet/pods/fa145b50-9bbb-4b69-94fa-c9b8c7e5fdf3/volumes"
Oct 30 04:09:52.027114 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:52.027075    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"]
Oct 30 04:09:52.036881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:52.036850    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8919eb-q2r99"]
Oct 30 04:09:53.698612 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:53.698580    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0134590-8140-487c-8286-3f663d7d942b" path="/var/lib/kubelet/pods/f0134590-8140-487c-8286-3f663d7d942b/volumes"
Oct 30 04:09:57.027535 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:57.027497    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"]
Oct 30 04:09:57.037208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:57.037177    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958b9133-8j2kd"]
Oct 30 04:09:57.698574 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:09:57.698535    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9206ed1-6957-4cea-954c-fe94b0a43d42" path="/var/lib/kubelet/pods/d9206ed1-6957-4cea-954c-fe94b0a43d42/volumes"
Oct 30 04:10:03.026454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:03.026402    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"]
Oct 30 04:10:03.036355 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:03.036327    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af3c4c3-7tqsm"]
Oct 30 04:10:03.697523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:03.697487    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4092798d-6c7a-4dd4-98e1-306a079f0ff6" path="/var/lib/kubelet/pods/4092798d-6c7a-4dd4-98e1-306a079f0ff6/volumes"
Oct 30 04:10:06.947941 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:06.947906    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"]
Oct 30 04:10:06.948641 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:06.948608    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" containerID="cri-o://4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c" gracePeriod=30
Oct 30 04:10:08.026948 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:08.026903    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"]
Oct 30 04:10:08.036528 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:08.036501    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05787b98-8z5gh"]
Oct 30 04:10:09.364664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:09.364629    4877 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:10:09.365102 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:09.364698    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:09.697675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:09.697594    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad31ddda-b88d-4deb-93c0-7fd22afe99e8" path="/var/lib/kubelet/pods/ad31ddda-b88d-4deb-93c0-7fd22afe99e8/volumes"
Oct 30 04:10:12.215468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.215395    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:10:12.298064 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.298018    4877 generic.go:356] "Generic (PLEG): container finished" podID="f435747d-c138-430d-9a05-c3789c910abf" containerID="4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c" exitCode=0
Oct 30 04:10:12.298273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.298175    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" event={"ID":"f435747d-c138-430d-9a05-c3789c910abf","Type":"ContainerDied","Data":"4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c"}
Oct 30 04:10:12.298273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.298267    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz" event={"ID":"f435747d-c138-430d-9a05-c3789c910abf","Type":"ContainerDied","Data":"878032dc4b42948dd5ca232df14bdb62239eaa500144591e5cb54b3543397511"}
Oct 30 04:10:12.298405 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.298309    4877 scope.go:122] "RemoveContainer" containerID="4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c"
Oct 30 04:10:12.298552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.298534    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"
Oct 30 04:10:12.309402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.309374    4877 scope.go:122] "RemoveContainer" containerID="963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"
Oct 30 04:10:12.319331 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.319300    4877 scope.go:122] "RemoveContainer" containerID="4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c"
Oct 30 04:10:12.319634 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:12.319615    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c\": container with ID starting with 4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c not found: ID does not exist" containerID="4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c"
Oct 30 04:10:12.319701 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.319643    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c"} err="failed to get container status \"4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c\": rpc error: code = NotFound desc = could not find container \"4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c\": container with ID starting with 4fa16fd8a7917c28b64ea78d00f11fc6ea6d8b9624bf9b3a92acadfe8082c22c not found: ID does not exist"
Oct 30 04:10:12.319701 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.319661    4877 scope.go:122] "RemoveContainer" containerID="963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"
Oct 30 04:10:12.319918 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:12.319899    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da\": container with ID starting with 963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da not found: ID does not exist" containerID="963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"
Oct 30 04:10:12.319958 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.319925    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da"} err="failed to get container status \"963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da\": rpc error: code = NotFound desc = could not find container \"963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da\": container with ID starting with 963d0e498aac470bf888af13d8d533c29dfe5e9c78daa4c1399c475e016508da not found: ID does not exist"
Oct 30 04:10:12.324244 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324224    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324291    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324358    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324462 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324394    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324557 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324538    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324653 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324579    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324653 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324620    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324785 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324689    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324785 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324729    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4\") pod \"f435747d-c138-430d-9a05-c3789c910abf\" (UID: \"f435747d-c138-430d-9a05-c3789c910abf\") "
Oct 30 04:10:12.324943 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324910    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:12.325043 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.324944    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:12.325140 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325053    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:12.325399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325373    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:12.325615 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325497    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:12.325615 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325589    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.325777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325614    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f435747d-c138-430d-9a05-c3789c910abf-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.325777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325629    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.325777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.325644    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.326751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.326723    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:10:12.326858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.326760    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:12.327353 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.327330    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:12.327490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.327386    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4" pod "f435747d-c138-430d-9a05-c3789c910abf" (UID: "f435747d-c138-430d-9a05-c3789c910abf"). InnerVolumeSpecName "kube-api-access-qv9f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:12.426386 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.426351    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.426386 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.426378    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.426386 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.426388    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9f4\" (UniqueName: \"kubernetes.io/projected/f435747d-c138-430d-9a05-c3789c910abf-kube-api-access-qv9f4\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.426662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.426399    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f435747d-c138-430d-9a05-c3789c910abf-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.426662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.426407    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f435747d-c138-430d-9a05-c3789c910abf-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:12.641103 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.641071    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"]
Oct 30 04:10:12.644384 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:12.644357    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-w7skz"]
Oct 30 04:10:13.697692 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:13.697656    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f435747d-c138-430d-9a05-c3789c910abf" path="/var/lib/kubelet/pods/f435747d-c138-430d-9a05-c3789c910abf/volumes"
Oct 30 04:10:14.028697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:14.028651    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"]
Oct 30 04:10:14.036549 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:14.036518    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307dddd6-mbbfz"]
Oct 30 04:10:15.697922 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:15.697888    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376e4d9a-84f8-4af3-b70e-dfaf9e43dfde" path="/var/lib/kubelet/pods/376e4d9a-84f8-4af3-b70e-dfaf9e43dfde/volumes"
Oct 30 04:10:16.228881 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:16.228842    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:10:19.027020 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:19.026979    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"]
Oct 30 04:10:19.040938 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:19.040902    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf1edc9-6kd6j"]
Oct 30 04:10:19.699296 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:19.699251    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fe115c-36f5-4e69-b643-09cf657a36c8" path="/var/lib/kubelet/pods/c1fe115c-36f5-4e69-b643-09cf657a36c8/volumes"
Oct 30 04:10:22.077186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:22.077150    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"]
Oct 30 04:10:22.077969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:22.077930    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" containerID="cri-o://a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d" gracePeriod=30
Oct 30 04:10:23.970091 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:23.970040    4877 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:10:23.970570 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:23.970120    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:27.343220 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.343152    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 04:10:27.416816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.416774    4877 generic.go:356] "Generic (PLEG): container finished" podID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerID="a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d" exitCode=0
Oct 30 04:10:27.417025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.416854    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"
Oct 30 04:10:27.417025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.416928    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" event={"ID":"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9","Type":"ContainerDied","Data":"a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d"}
Oct 30 04:10:27.417025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.416981    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q" event={"ID":"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9","Type":"ContainerDied","Data":"5a495c81058ea786fac1b9af40f01d54ff13e4be089bb4ae83f7b7d97019cf28"}
Oct 30 04:10:27.417025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.417017    4877 scope.go:122] "RemoveContainer" containerID="a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d"
Oct 30 04:10:27.431713 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.431689    4877 scope.go:122] "RemoveContainer" containerID="8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"
Oct 30 04:10:27.441803 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.441782    4877 scope.go:122] "RemoveContainer" containerID="a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d"
Oct 30 04:10:27.442086 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:27.442057    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d\": container with ID starting with a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d not found: ID does not exist" containerID="a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d"
Oct 30 04:10:27.442156 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.442095    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d"} err="failed to get container status \"a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d\": rpc error: code = NotFound desc = could not find container \"a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d\": container with ID starting with a6ade70dd44cbc7543eb72271d80f3ece0d52870ece9ddd0b8e6832e7b7c509d not found: ID does not exist"
Oct 30 04:10:27.442156 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.442114    4877 scope.go:122] "RemoveContainer" containerID="8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"
Oct 30 04:10:27.442383 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:27.442364    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6\": container with ID starting with 8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6 not found: ID does not exist" containerID="8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"
Oct 30 04:10:27.442452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.442389    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6"} err="failed to get container status \"8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6\": rpc error: code = NotFound desc = could not find container \"8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6\": container with ID starting with 8b08b6a12817a1b7b152cbe510de4ac3ac81d23cf18ae23093009515168bb7c6 not found: ID does not exist"
Oct 30 04:10:27.504458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504413    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.504572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504498    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.504572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504568    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.504693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504599    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.504693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504636    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.504693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504651    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:27.504693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504676    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.504999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.504963    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:27.505151 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.505108    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:27.505373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.505355    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.505564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.505541    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:27.505672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.505600    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.505737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.505680    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6\") pod \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\" (UID: \"828f9787-d0fc-4aad-8c2d-04f2a21cf3a9\") "
Oct 30 04:10:27.505865 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.505846    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:27.506071 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.506051    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.506172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.506076    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.506172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.506091    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.506172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.506103    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.506172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.506115    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.507093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.507069    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:27.507315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.507283    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:27.507315 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.507302    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:10:27.508162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.508137    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6" pod "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" (UID: "828f9787-d0fc-4aad-8c2d-04f2a21cf3a9"). InnerVolumeSpecName "kube-api-access-4thd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:27.606869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.606779    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thd6\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-kube-api-access-4thd6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.606869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.606823    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.606869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.606834    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.606869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.606842    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:27.752784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.752753    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"]
Oct 30 04:10:27.762189 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:27.762160    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-stp5q"]
Oct 30 04:10:29.697329 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:29.697289    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" path="/var/lib/kubelet/pods/828f9787-d0fc-4aad-8c2d-04f2a21cf3a9/volumes"
Oct 30 04:10:38.358819 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.358784    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"]
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359226    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1c572573-47ef-4b27-9e13-c1a561d30b9f" containerName="revoke-keys"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359237    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c572573-47ef-4b27-9e13-c1a561d30b9f" containerName="revoke-keys"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359248    4877 cpu_manager.go:414] "containerMap: removing container" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359253    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359281    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359287    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359299    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy"
Oct 30 04:10:38.359310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359304    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy"
Oct 30 04:10:38.359762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359376    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy"
Oct 30 04:10:38.359762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359384    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy"
Oct 30 04:10:38.359762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359392    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f435747d-c138-430d-9a05-c3789c910abf" containerName="istio-proxy"
Oct 30 04:10:38.359762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.359402    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c572573-47ef-4b27-9e13-c1a561d30b9f" containerName="revoke-keys"
Oct 30 04:10:38.362481 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.362464    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.387697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.387672    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:10:38.387843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.387708    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:10:38.392937 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.392910    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"]
Oct 30 04:10:38.428070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.428039    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzvt\" (UniqueName: \"kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.428210 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.428079    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.428260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.428208    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.528880 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.528840    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzvt\" (UniqueName: \"kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.529088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.528880    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.529088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.529012    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.529630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.529605    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.531291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.531268    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.545525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.545504    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzvt\" (UniqueName: \"kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt\") pod \"maas-api-revoke-sub-bc9eddf8-lrhjz\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.677795 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.677717    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:38.981604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:38.981565    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"]
Oct 30 04:10:38.982520 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:10:38.982492    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bce459d_bbe2_4433_ab19_bd65c9fbec8a.slice/crio-f51ae372ad7a99aabf27672a521c3be6531d7dbbf0597e36ad10c18de9b959b4 WatchSource:0}: Error finding container f51ae372ad7a99aabf27672a521c3be6531d7dbbf0597e36ad10c18de9b959b4: Status 404 returned error can't find the container with id f51ae372ad7a99aabf27672a521c3be6531d7dbbf0597e36ad10c18de9b959b4
Oct 30 04:10:39.518876 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:39.518836    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz" event={"ID":"9bce459d-bbe2-4433-ab19-bd65c9fbec8a","Type":"ContainerStarted","Data":"dbd1c816a9e428c8639299cf0fa7a87d305e4922bf61f12dd8eee0e17df525ee"}
Oct 30 04:10:39.519360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:39.518889    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz" event={"ID":"9bce459d-bbe2-4433-ab19-bd65c9fbec8a","Type":"ContainerStarted","Data":"f51ae372ad7a99aabf27672a521c3be6531d7dbbf0597e36ad10c18de9b959b4"}
Oct 30 04:10:39.555261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:39.555160    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz" podStartSLOduration=1.555140953 podStartE2EDuration="1.555140953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:39.551834167 +0000 UTC m=+4067.980723603" watchObservedRunningTime="2026-10-30 04:10:39.555140953 +0000 UTC m=+4067.984030371"
Oct 30 04:10:40.527727 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:40.527686    4877 generic.go:356] "Generic (PLEG): container finished" podID="9bce459d-bbe2-4433-ab19-bd65c9fbec8a" containerID="dbd1c816a9e428c8639299cf0fa7a87d305e4922bf61f12dd8eee0e17df525ee" exitCode=0
Oct 30 04:10:40.528129 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:40.527765    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz" event={"ID":"9bce459d-bbe2-4433-ab19-bd65c9fbec8a","Type":"ContainerDied","Data":"dbd1c816a9e428c8639299cf0fa7a87d305e4922bf61f12dd8eee0e17df525ee"}
Oct 30 04:10:41.679728 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.679702    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:41.762951 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.762920    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca\") pod \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") "
Oct 30 04:10:41.763351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.763025    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt\" (UniqueName: \"kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt\") pod \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") "
Oct 30 04:10:41.763351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.763053    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca\") pod \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\" (UID: \"9bce459d-bbe2-4433-ab19-bd65c9fbec8a\") "
Oct 30 04:10:41.763553 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.763448    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca" pod "9bce459d-bbe2-4433-ab19-bd65c9fbec8a" (UID: "9bce459d-bbe2-4433-ab19-bd65c9fbec8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:41.765158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.765136    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca" pod "9bce459d-bbe2-4433-ab19-bd65c9fbec8a" (UID: "9bce459d-bbe2-4433-ab19-bd65c9fbec8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:41.765234 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.765166    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt" pod "9bce459d-bbe2-4433-ab19-bd65c9fbec8a" (UID: "9bce459d-bbe2-4433-ab19-bd65c9fbec8a"). InnerVolumeSpecName "kube-api-access-6pzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:41.863969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.863886    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:41.863969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.863919    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzvt\" (UniqueName: \"kubernetes.io/projected/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-kube-api-access-6pzvt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:41.863969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:41.863931    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bce459d-bbe2-4433-ab19-bd65c9fbec8a-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:42.542891 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:42.542860    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"
Oct 30 04:10:42.543097 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:42.542860    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz" event={"ID":"9bce459d-bbe2-4433-ab19-bd65c9fbec8a","Type":"ContainerDied","Data":"f51ae372ad7a99aabf27672a521c3be6531d7dbbf0597e36ad10c18de9b959b4"}
Oct 30 04:10:42.543097 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:42.542988    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f51ae372ad7a99aabf27672a521c3be6531d7dbbf0597e36ad10c18de9b959b4"
Oct 30 04:10:42.828521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:42.828442    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"]
Oct 30 04:10:46.229543 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:46.229499    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:10:48.973380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:48.973340    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"]
Oct 30 04:10:49.008085 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.008048    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"]
Oct 30 04:10:49.167369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.167331    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"]
Oct 30 04:10:49.168063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.168037    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9bce459d-bbe2-4433-ab19-bd65c9fbec8a" containerName="revoke-keys"
Oct 30 04:10:49.168063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.168059    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bce459d-bbe2-4433-ab19-bd65c9fbec8a" containerName="revoke-keys"
Oct 30 04:10:49.168063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.168068    4877 cpu_manager.go:414] "containerMap: removing container" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy"
Oct 30 04:10:49.168340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.168073    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy"
Oct 30 04:10:49.168340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.168175    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bce459d-bbe2-4433-ab19-bd65c9fbec8a" containerName="revoke-keys"
Oct 30 04:10:49.168340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.168189    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="828f9787-d0fc-4aad-8c2d-04f2a21cf3a9" containerName="istio-proxy"
Oct 30 04:10:49.171757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.171736    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.186079 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.186053    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:10:49.186366 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.186343    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:10:49.189858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.189830    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"]
Oct 30 04:10:49.243448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.243326    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhw9l\" (UniqueName: \"kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.243648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.243603    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.243733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.243671    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.344831 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.344796    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.345058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.344865    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.345058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.344992    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhw9l\" (UniqueName: \"kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.345697 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.345672    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.347267 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.347241    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.362027 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.362002    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhw9l\" (UniqueName: \"kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l\") pod \"maas-api-revoke-sub-71b487de-c9lc7\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.485337 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.485303    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:49.792665 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:49.792637    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"]
Oct 30 04:10:49.795438 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:10:49.795392    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00db8e5_5be4_40cc_9cdb_ea14525dffd2.slice/crio-abee097d27b8dd5e3b39aebdc2b633a7073f2fc3630b6d61270665fb3869362d WatchSource:0}: Error finding container abee097d27b8dd5e3b39aebdc2b633a7073f2fc3630b6d61270665fb3869362d: Status 404 returned error can't find the container with id abee097d27b8dd5e3b39aebdc2b633a7073f2fc3630b6d61270665fb3869362d
Oct 30 04:10:50.642974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:50.642919    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7" event={"ID":"a00db8e5-5be4-40cc-9cdb-ea14525dffd2","Type":"ContainerStarted","Data":"89404ed884f62c3e5f40d0f01b615ad2f5b6d386cb8097c0e8b55a6650172f88"}
Oct 30 04:10:50.643471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:50.643015    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7" event={"ID":"a00db8e5-5be4-40cc-9cdb-ea14525dffd2","Type":"ContainerStarted","Data":"abee097d27b8dd5e3b39aebdc2b633a7073f2fc3630b6d61270665fb3869362d"}
Oct 30 04:10:50.681225 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:50.681171    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7" podStartSLOduration=1.6811577180000001 podStartE2EDuration="1.681157718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:50.678036979 +0000 UTC m=+4079.106926422" watchObservedRunningTime="2026-10-30 04:10:50.681157718 +0000 UTC m=+4079.110047115"
Oct 30 04:10:51.652172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:51.652136    4877 generic.go:356] "Generic (PLEG): container finished" podID="a00db8e5-5be4-40cc-9cdb-ea14525dffd2" containerID="89404ed884f62c3e5f40d0f01b615ad2f5b6d386cb8097c0e8b55a6650172f88" exitCode=0
Oct 30 04:10:51.652617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:51.652207    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7" event={"ID":"a00db8e5-5be4-40cc-9cdb-ea14525dffd2","Type":"ContainerDied","Data":"89404ed884f62c3e5f40d0f01b615ad2f5b6d386cb8097c0e8b55a6650172f88"}
Oct 30 04:10:52.827925 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.827892    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:52.884057 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.884024    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca\") pod \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") "
Oct 30 04:10:52.884267 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.884118    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l\" (UniqueName: \"kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l\") pod \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") "
Oct 30 04:10:52.884267 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.884173    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca\") pod \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\" (UID: \"a00db8e5-5be4-40cc-9cdb-ea14525dffd2\") "
Oct 30 04:10:52.884610 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.884564    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca" pod "a00db8e5-5be4-40cc-9cdb-ea14525dffd2" (UID: "a00db8e5-5be4-40cc-9cdb-ea14525dffd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:52.886351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.886320    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca" pod "a00db8e5-5be4-40cc-9cdb-ea14525dffd2" (UID: "a00db8e5-5be4-40cc-9cdb-ea14525dffd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:52.886962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.886932    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l" pod "a00db8e5-5be4-40cc-9cdb-ea14525dffd2" (UID: "a00db8e5-5be4-40cc-9cdb-ea14525dffd2"). InnerVolumeSpecName "kube-api-access-mhw9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:52.985551 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.985456    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:52.985551 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.985484    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhw9l\" (UniqueName: \"kubernetes.io/projected/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-kube-api-access-mhw9l\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:52.985551 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:52.985495    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00db8e5-5be4-40cc-9cdb-ea14525dffd2-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:53.671703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:53.671662    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7" event={"ID":"a00db8e5-5be4-40cc-9cdb-ea14525dffd2","Type":"ContainerDied","Data":"abee097d27b8dd5e3b39aebdc2b633a7073f2fc3630b6d61270665fb3869362d"}
Oct 30 04:10:53.671703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:53.671695    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"
Oct 30 04:10:53.671933 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:53.671725    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abee097d27b8dd5e3b39aebdc2b633a7073f2fc3630b6d61270665fb3869362d"
Oct 30 04:10:53.974306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:53.974196    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" containerName="payload-processing" containerID="cri-o://1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6" gracePeriod=25
Oct 30 04:10:54.008899 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:54.008861    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerName="payload-pre-processing" containerID="cri-o://65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c" gracePeriod=25
Oct 30 04:10:57.629821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:57.629784    4877 scope.go:122] "RemoveContainer" containerID="451274e18b8d0549d5a68fc8da14e6c84c9faf43ada7fef9ba969bfe2d32a567"
Oct 30 04:10:57.639887 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:57.639867    4877 scope.go:122] "RemoveContainer" containerID="be7732ab9f2b7e96be3527fb89eb4c2fec556899257aeb47a4b519e68deaf4cf"
Oct 30 04:10:57.648711 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:57.648687    4877 scope.go:122] "RemoveContainer" containerID="e1d990b5bb04d9cb96e3a485163360531dfc942431a9236deeb875910a0cda88"
Oct 30 04:10:57.657296 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:57.657270    4877 scope.go:122] "RemoveContainer" containerID="d4593f59fe4453123833c4570a6879cb13d167c9fc1033ad4c255aa4c27de8fb"
Oct 30 04:10:57.849033 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:57.848992    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" containerID="cri-o://52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4" gracePeriod=45
Oct 30 04:10:57.852751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:57.852711    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": dial tcp 10.133.0.96:8000: connect: connection refused"
Oct 30 04:10:58.095116 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.095086    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 04:10:58.241669 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241586    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home\") pod \"8e247fa9-29b7-4f01-891a-ea4bde141301\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") "
Oct 30 04:10:58.241856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241680    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache\") pod \"8e247fa9-29b7-4f01-891a-ea4bde141301\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") "
Oct 30 04:10:58.241856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241721    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9\" (UniqueName: \"kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9\") pod \"8e247fa9-29b7-4f01-891a-ea4bde141301\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") "
Oct 30 04:10:58.241856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241757    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs\") pod \"8e247fa9-29b7-4f01-891a-ea4bde141301\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") "
Oct 30 04:10:58.241856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241793    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm\") pod \"8e247fa9-29b7-4f01-891a-ea4bde141301\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") "
Oct 30 04:10:58.241856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241822    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir\") pod \"8e247fa9-29b7-4f01-891a-ea4bde141301\" (UID: \"8e247fa9-29b7-4f01-891a-ea4bde141301\") "
Oct 30 04:10:58.242168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241930    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home" pod "8e247fa9-29b7-4f01-891a-ea4bde141301" (UID: "8e247fa9-29b7-4f01-891a-ea4bde141301"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:58.242168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.241956    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache" pod "8e247fa9-29b7-4f01-891a-ea4bde141301" (UID: "8e247fa9-29b7-4f01-891a-ea4bde141301"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:58.242295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.242218    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir" pod "8e247fa9-29b7-4f01-891a-ea4bde141301" (UID: "8e247fa9-29b7-4f01-891a-ea4bde141301"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:58.242361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.242289    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:58.242361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.242310    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:58.244098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.244070    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9" pod "8e247fa9-29b7-4f01-891a-ea4bde141301" (UID: "8e247fa9-29b7-4f01-891a-ea4bde141301"). InnerVolumeSpecName "kube-api-access-b5lm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:58.244294 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.244276    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs" pod "8e247fa9-29b7-4f01-891a-ea4bde141301" (UID: "8e247fa9-29b7-4f01-891a-ea4bde141301"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:58.244364 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.244280    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm" pod "8e247fa9-29b7-4f01-891a-ea4bde141301" (UID: "8e247fa9-29b7-4f01-891a-ea4bde141301"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:58.343030 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.342989    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lm9\" (UniqueName: \"kubernetes.io/projected/8e247fa9-29b7-4f01-891a-ea4bde141301-kube-api-access-b5lm9\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:58.343030 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.343023    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e247fa9-29b7-4f01-891a-ea4bde141301-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:58.343030 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.343036    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:58.343319 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.343046    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e247fa9-29b7-4f01-891a-ea4bde141301-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:10:58.706963 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.706926    4877 generic.go:356] "Generic (PLEG): container finished" podID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerID="52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4" exitCode=0
Oct 30 04:10:58.707392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.706974    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"
Oct 30 04:10:58.707392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.707010    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" event={"ID":"8e247fa9-29b7-4f01-891a-ea4bde141301","Type":"ContainerDied","Data":"52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4"}
Oct 30 04:10:58.707392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.707064    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75" event={"ID":"8e247fa9-29b7-4f01-891a-ea4bde141301","Type":"ContainerDied","Data":"2da9ff843de51af1acc491950ab8705fd525c56b0fedec7e98d6eaab6f534c32"}
Oct 30 04:10:58.707392 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.707082    4877 scope.go:122] "RemoveContainer" containerID="52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4"
Oct 30 04:10:58.718151 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.718135    4877 scope.go:122] "RemoveContainer" containerID="52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4"
Oct 30 04:10:58.718446 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:10:58.718407    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4\": container with ID starting with 52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4 not found: ID does not exist" containerID="52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4"
Oct 30 04:10:58.718507 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.718446    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4"} err="failed to get container status \"52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4\": rpc error: code = NotFound desc = could not find container \"52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4\": container with ID starting with 52c7e1db6579303f645c42bf1184c688a6fa4e7155388a95e54256f8d544c9d4 not found: ID does not exist"
Oct 30 04:10:58.752026 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.751998    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"]
Oct 30 04:10:58.761499 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:58.761476    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2d5b09/distinct-w2-2d5b09-kserve-797b7dbd77-m7n75"]
Oct 30 04:10:59.697780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:10:59.697748    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" path="/var/lib/kubelet/pods/8e247fa9-29b7-4f01-891a-ea4bde141301/volumes"
Oct 30 04:11:00.368734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:00.368690    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:11:00.368734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:00.368718    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:11:04.275177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:04.275101    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b66c856-5qx55_f34c49c1-2d78-4b79-8505-f68314e4d4b9/authorino/0.log"
Oct 30 04:11:04.534997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:04.534913    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-npmx5_297ec4ff-cbaf-48ea-a5cd-fab91d07a75e/manager/0.log"
Oct 30 04:11:04.766929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:04.766897    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gggrh_713ca209-955a-44dd-b634-18e58ef6735b/manager/0.log"
Oct 30 04:11:04.982068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:04.981991    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8m78v_490cd1fd-4982-4de2-824f-1a3f359dc24d/kuadrant-console-plugin/0.log"
Oct 30 04:11:05.228462 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:05.228414    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-txkl6_adde55dc-547d-42d9-9cc6-7e281e545561/manager/0.log"
Oct 30 04:11:05.705755 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:05.705724    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rcpz6_66695834-c9cd-430e-946c-aa57ae794bca/manager/0.log"
Oct 30 04:11:09.672886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:09.672850    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:11:09.946834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:09.946761    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:11:10.368558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:10.368519    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:11:10.368558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:10.368542    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:11:14.268133 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.268108    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 04:11:14.296824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.296800    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 04:11:14.414835 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.414801    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume\") pod \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") "
Oct 30 04:11:14.415038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.414885    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume\") pod \"5448a7de-48a8-4a48-9674-c113cbf14868\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") "
Oct 30 04:11:14.415038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.414924    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq\" (UniqueName: \"kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq\") pod \"5448a7de-48a8-4a48-9674-c113cbf14868\" (UID: \"5448a7de-48a8-4a48-9674-c113cbf14868\") "
Oct 30 04:11:14.415038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.415012    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq\" (UniqueName: \"kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq\") pod \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\" (UID: \"6e19aa13-f266-4f60-b227-5c7b2ce8031a\") "
Oct 30 04:11:14.415256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.415225    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume" pod "6e19aa13-f266-4f60-b227-5c7b2ce8031a" (UID: "6e19aa13-f266-4f60-b227-5c7b2ce8031a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:11:14.415432 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.415320    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume" pod "5448a7de-48a8-4a48-9674-c113cbf14868" (UID: "5448a7de-48a8-4a48-9674-c113cbf14868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:11:14.417248 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.417224    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq" pod "5448a7de-48a8-4a48-9674-c113cbf14868" (UID: "5448a7de-48a8-4a48-9674-c113cbf14868"). InnerVolumeSpecName "kube-api-access-gw9fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:11:14.417248 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.417240    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq" pod "6e19aa13-f266-4f60-b227-5c7b2ce8031a" (UID: "6e19aa13-f266-4f60-b227-5c7b2ce8031a"). InnerVolumeSpecName "kube-api-access-54plq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:11:14.516895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.516810    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e19aa13-f266-4f60-b227-5c7b2ce8031a-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:14.516895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.516842    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5448a7de-48a8-4a48-9674-c113cbf14868-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:14.516895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.516851    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw9fq\" (UniqueName: \"kubernetes.io/projected/5448a7de-48a8-4a48-9674-c113cbf14868-kube-api-access-gw9fq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:14.516895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.516860    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54plq\" (UniqueName: \"kubernetes.io/projected/6e19aa13-f266-4f60-b227-5c7b2ce8031a-kube-api-access-54plq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:14.819947 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.819905    4877 generic.go:356] "Generic (PLEG): container finished" podID="5448a7de-48a8-4a48-9674-c113cbf14868" containerID="1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6" exitCode=0
Oct 30 04:11:14.819947 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.819945    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"
Oct 30 04:11:14.820230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.819984    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" event={"ID":"5448a7de-48a8-4a48-9674-c113cbf14868","Type":"ContainerDied","Data":"1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6"}
Oct 30 04:11:14.820230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.820041    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h" event={"ID":"5448a7de-48a8-4a48-9674-c113cbf14868","Type":"ContainerDied","Data":"8247f6f93da3773f16543bf9d7dea21e0f8a9a8ae80dfe40a103dd904c1d1782"}
Oct 30 04:11:14.820230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.820056    4877 scope.go:122] "RemoveContainer" containerID="1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6"
Oct 30 04:11:14.821664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.821626    4877 generic.go:356] "Generic (PLEG): container finished" podID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerID="65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c" exitCode=0
Oct 30 04:11:14.821799 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.821658    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" event={"ID":"6e19aa13-f266-4f60-b227-5c7b2ce8031a","Type":"ContainerDied","Data":"65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c"}
Oct 30 04:11:14.821799 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.821711    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz" event={"ID":"6e19aa13-f266-4f60-b227-5c7b2ce8031a","Type":"ContainerDied","Data":"5fdadf837269f51d150fe28238e0fb7c7d306092f2cdbfed96f8de77dc799a50"}
Oct 30 04:11:14.821799 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.821667    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"
Oct 30 04:11:14.832153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.832121    4877 scope.go:122] "RemoveContainer" containerID="1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6"
Oct 30 04:11:14.832675 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:11:14.832515    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6\": container with ID starting with 1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6 not found: ID does not exist" containerID="1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6"
Oct 30 04:11:14.832675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.832551    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6"} err="failed to get container status \"1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6\": rpc error: code = NotFound desc = could not find container \"1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6\": container with ID starting with 1bede58b55db1c57113466087f834082464a3687fc47edbff45233a2ec6f55d6 not found: ID does not exist"
Oct 30 04:11:14.832675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.832576    4877 scope.go:122] "RemoveContainer" containerID="65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c"
Oct 30 04:11:14.842385 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.842365    4877 scope.go:122] "RemoveContainer" containerID="65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c"
Oct 30 04:11:14.843196 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:11:14.843177    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c\": container with ID starting with 65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c not found: ID does not exist" containerID="65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c"
Oct 30 04:11:14.843283 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.843202    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c"} err="failed to get container status \"65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c\": rpc error: code = NotFound desc = could not find container \"65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c\": container with ID starting with 65fc3f8d5ea380c9325368e4322d1e9e09a83b0b19172d1978098a5598e7477c not found: ID does not exist"
Oct 30 04:11:14.877949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.877922    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"]
Oct 30 04:11:14.881005 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.880979    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2d5b09-5c64bbc7f5-qzskz"]
Oct 30 04:11:14.917246 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.917218    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"]
Oct 30 04:11:14.922734 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:14.922710    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2d5b09-5f84fd87fd-bjq2h"]
Oct 30 04:11:15.697258 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:15.697215    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" path="/var/lib/kubelet/pods/5448a7de-48a8-4a48-9674-c113cbf14868/volumes"
Oct 30 04:11:15.697844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:15.697820    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" path="/var/lib/kubelet/pods/6e19aa13-f266-4f60-b227-5c7b2ce8031a/volumes"
Oct 30 04:11:16.228656 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:11:16.228621    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:11:30.026762 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:30.026728    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"]
Oct 30 04:11:30.036061 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:30.036038    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cfa8300-dxb5z"]
Oct 30 04:11:31.699483 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:31.699448    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e" path="/var/lib/kubelet/pods/0d0f14d5-80c7-466e-b0e8-cb52a8f5bc7e/volumes"
Oct 30 04:11:37.741668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.741633    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"]
Oct 30 04:11:37.742077 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742064    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a00db8e5-5be4-40cc-9cdb-ea14525dffd2" containerName="revoke-keys"
Oct 30 04:11:37.742127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742079    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00db8e5-5be4-40cc-9cdb-ea14525dffd2" containerName="revoke-keys"
Oct 30 04:11:37.742127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742095    4877 cpu_manager.go:414] "containerMap: removing container" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main"
Oct 30 04:11:37.742127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742101    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main"
Oct 30 04:11:37.742127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742115    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerName="payload-pre-processing"
Oct 30 04:11:37.742127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742120    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerName="payload-pre-processing"
Oct 30 04:11:37.742304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742142    4877 cpu_manager.go:414] "containerMap: removing container" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" containerName="payload-processing"
Oct 30 04:11:37.742304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742147    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" containerName="payload-processing"
Oct 30 04:11:37.742304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742210    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="5448a7de-48a8-4a48-9674-c113cbf14868" containerName="payload-processing"
Oct 30 04:11:37.742304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742219    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00db8e5-5be4-40cc-9cdb-ea14525dffd2" containerName="revoke-keys"
Oct 30 04:11:37.742304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742226    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e247fa9-29b7-4f01-891a-ea4bde141301" containerName="main"
Oct 30 04:11:37.742304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.742234    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e19aa13-f266-4f60-b227-5c7b2ce8031a" containerName="payload-pre-processing"
Oct 30 04:11:37.745662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.745643    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.767079 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.767052    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"]
Oct 30 04:11:37.773434 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.773395    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:11:37.773633 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.773611    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\""
Oct 30 04:11:37.847489 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.847452    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vwv\" (UniqueName: \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.847664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.847508    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.847664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.847590    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.948692 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.948650    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.948908 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.948793    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vwv\" (UniqueName: \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.948908 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.948855    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.949479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.949414    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.951176 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:37.951152    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:37.971662 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:11:37.971640    4877 projected.go:196] Error preparing data for projected volume kube-api-access-k8vwv for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 04:11:37.971781 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:11:37.971727    4877 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv podName:c21e112b-f3c3-4888-bb2e-c12d0d8441c0 nodeName:}" failed. No retries permitted until 2026-10-30 04:11:38.471705456 +0000 UTC m=+4126.900594858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k8vwv" (UniqueName: "kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv") pod "maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn" (UID: "c21e112b-f3c3-4888-bb2e-c12d0d8441c0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 04:11:38.555853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:38.555802    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vwv\" (UniqueName: \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:38.572197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:38.572165    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vwv\" (UniqueName: \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\") pod \"maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:38.658945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:38.658911    4877 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn" secret="" err="secret \"maas-api-cleanup-dockercfg-nvf2f\" not found"
Oct 30 04:11:38.659134 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:38.658976    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:38.959668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:38.959638    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"]
Oct 30 04:11:38.962206 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:11:38.962171    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21e112b_f3c3_4888_bb2e_c12d0d8441c0.slice/crio-5d8acaabd1a372d2eac5a74f0220ea0e476ab38469b026d04e1e94654eec6d97 WatchSource:0}: Error finding container 5d8acaabd1a372d2eac5a74f0220ea0e476ab38469b026d04e1e94654eec6d97: Status 404 returned error can't find the container with id 5d8acaabd1a372d2eac5a74f0220ea0e476ab38469b026d04e1e94654eec6d97
Oct 30 04:11:38.989740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:38.989718    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn" event={"ID":"c21e112b-f3c3-4888-bb2e-c12d0d8441c0","Type":"ContainerStarted","Data":"5d8acaabd1a372d2eac5a74f0220ea0e476ab38469b026d04e1e94654eec6d97"}
Oct 30 04:11:39.998227 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:39.998192    4877 generic.go:356] "Generic (PLEG): container finished" podID="c21e112b-f3c3-4888-bb2e-c12d0d8441c0" containerID="8482bb7d45b53d856d24a231e438b46f41d6f0a67a056421adce83322683b976" exitCode=0
Oct 30 04:11:39.998706 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:39.998278    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn" event={"ID":"c21e112b-f3c3-4888-bb2e-c12d0d8441c0","Type":"ContainerDied","Data":"8482bb7d45b53d856d24a231e438b46f41d6f0a67a056421adce83322683b976"}
Oct 30 04:11:41.035272 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.035234    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"]
Oct 30 04:11:41.044602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.044571    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95895851-ml9hr"]
Oct 30 04:11:41.149079 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.149051    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:41.280595 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.280561    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca\") pod \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") "
Oct 30 04:11:41.280784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.280624    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\" (UniqueName: \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\") pod \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") "
Oct 30 04:11:41.280784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.280656    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca\") pod \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\" (UID: \"c21e112b-f3c3-4888-bb2e-c12d0d8441c0\") "
Oct 30 04:11:41.280975 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.280950    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca" pod "c21e112b-f3c3-4888-bb2e-c12d0d8441c0" (UID: "c21e112b-f3c3-4888-bb2e-c12d0d8441c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:11:41.282866 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.282830    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca" pod "c21e112b-f3c3-4888-bb2e-c12d0d8441c0" (UID: "c21e112b-f3c3-4888-bb2e-c12d0d8441c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:11:41.282866 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.282846    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv" pod "c21e112b-f3c3-4888-bb2e-c12d0d8441c0" (UID: "c21e112b-f3c3-4888-bb2e-c12d0d8441c0"). InnerVolumeSpecName "kube-api-access-k8vwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:11:41.381630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.381597    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:41.381630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.381627    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vwv\" (UniqueName: \"kubernetes.io/projected/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-kube-api-access-k8vwv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:41.381630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.381638    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21e112b-f3c3-4888-bb2e-c12d0d8441c0-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:11:41.696892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:41.696804    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a87329d-e60b-404a-95d2-b32e6fc6001a" path="/var/lib/kubelet/pods/6a87329d-e60b-404a-95d2-b32e6fc6001a/volumes"
Oct 30 04:11:42.012718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:42.012634    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"
Oct 30 04:11:42.012718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:42.012670    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn" event={"ID":"c21e112b-f3c3-4888-bb2e-c12d0d8441c0","Type":"ContainerDied","Data":"5d8acaabd1a372d2eac5a74f0220ea0e476ab38469b026d04e1e94654eec6d97"}
Oct 30 04:11:42.012718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:42.012716    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8acaabd1a372d2eac5a74f0220ea0e476ab38469b026d04e1e94654eec6d97"
Oct 30 04:11:46.229124 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:11:46.229076    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:11:50.750694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:50.750655    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"]
Oct 30 04:11:50.759724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:50.759697    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e1eb434-7fprn"]
Oct 30 04:11:51.699327 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:51.699291    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21e112b-f3c3-4888-bb2e-c12d0d8441c0" path="/var/lib/kubelet/pods/c21e112b-f3c3-4888-bb2e-c12d0d8441c0/volumes"
Oct 30 04:11:57.706722 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:57.706689    4877 scope.go:122] "RemoveContainer" containerID="ab45565b59916bb0600428f0546b6e49477ca1b0dfe92bc39bba8e9636164fa2"
Oct 30 04:11:57.717037 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:57.717006    4877 scope.go:122] "RemoveContainer" containerID="58c5537128d13092a5f506f6dfd80c6f68efa2b95e36f901ae2484e0e49895fc"
Oct 30 04:11:57.725629 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:57.725593    4877 scope.go:122] "RemoveContainer" containerID="5da8821a945176fd67dac644e29af7a2a933ab2d51cc0ac13be06f1a41626075"
Oct 30 04:11:57.736477 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:57.736454    4877 scope.go:122] "RemoveContainer" containerID="fd64196683b0ebe6139b781dcb85301ba0db56c9147e1afa855e8ae517c100a6"
Oct 30 04:11:59.027652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:59.027615    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"]
Oct 30 04:11:59.037782 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:59.037749    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f7cacc-4wstn"]
Oct 30 04:11:59.697881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:11:59.697841    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab" path="/var/lib/kubelet/pods/713d5b0b-8b85-4ff9-8bc9-4e391d2aa1ab/volumes"
Oct 30 04:12:10.036277 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:10.036235    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"]
Oct 30 04:12:10.045833 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:10.045807    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72ea379-v8tc6"]
Oct 30 04:12:11.696961 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:11.696918    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a46e90-0bd9-4f85-ada5-cae7c03b074b" path="/var/lib/kubelet/pods/65a46e90-0bd9-4f85-ada5-cae7c03b074b/volumes"
Oct 30 04:12:16.228673 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:12:16.228635    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:12:17.983970 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:17.983932    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"]
Oct 30 04:12:17.984375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:17.984366    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c21e112b-f3c3-4888-bb2e-c12d0d8441c0" containerName="revoke-keys"
Oct 30 04:12:17.984437 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:17.984378    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21e112b-f3c3-4888-bb2e-c12d0d8441c0" containerName="revoke-keys"
Oct 30 04:12:17.984499 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:17.984488    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21e112b-f3c3-4888-bb2e-c12d0d8441c0" containerName="revoke-keys"
Oct 30 04:12:17.987910 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:17.987892    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.013394 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.013364    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:12:18.013588 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.013395    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:12:18.017321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.017271    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"]
Oct 30 04:12:18.027565 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.027531    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"]
Oct 30 04:12:18.033373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.033342    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-151512ce-p9fc8"]
Oct 30 04:12:18.068559 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.068527    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.068688 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.068594    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.068737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.068679    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g7h\" (UniqueName: \"kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.169816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.169778    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.170021 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.169827    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g7h\" (UniqueName: \"kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.170021 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.169889    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.170476 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.170449    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.172218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.172195    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.187316 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.187287    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g7h\" (UniqueName: \"kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h\") pod \"maas-api-revoke-sub-85237e45-xtdtz\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.301045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.301013    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:18.601872 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:18.601705    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"]
Oct 30 04:12:18.605345 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:12:18.605315    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff56652_26ec_4dc5_918d_e20a420a307d.slice/crio-61a1849af1235c190a27938598d4c82cb3b536a0192dbd7234c88a26e1a992b8 WatchSource:0}: Error finding container 61a1849af1235c190a27938598d4c82cb3b536a0192dbd7234c88a26e1a992b8: Status 404 returned error can't find the container with id 61a1849af1235c190a27938598d4c82cb3b536a0192dbd7234c88a26e1a992b8
Oct 30 04:12:19.262336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:19.262242    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz" event={"ID":"2ff56652-26ec-4dc5-918d-e20a420a307d","Type":"ContainerStarted","Data":"7e79a9ca7c0d4e587a0315996329f0d832791291916e51be9d1fefb73c3926fa"}
Oct 30 04:12:19.262336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:19.262298    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz" event={"ID":"2ff56652-26ec-4dc5-918d-e20a420a307d","Type":"ContainerStarted","Data":"61a1849af1235c190a27938598d4c82cb3b536a0192dbd7234c88a26e1a992b8"}
Oct 30 04:12:19.302715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:19.302663    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz" podStartSLOduration=2.30265189 podStartE2EDuration="2.30265189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:19.299408674 +0000 UTC m=+4167.728298093" watchObservedRunningTime="2026-10-30 04:12:19.30265189 +0000 UTC m=+4167.731541308"
Oct 30 04:12:19.697749 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:19.697716    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e097a1b6-6520-4472-bddf-adfeb6171e79" path="/var/lib/kubelet/pods/e097a1b6-6520-4472-bddf-adfeb6171e79/volumes"
Oct 30 04:12:20.269571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:20.269537    4877 generic.go:356] "Generic (PLEG): container finished" podID="2ff56652-26ec-4dc5-918d-e20a420a307d" containerID="7e79a9ca7c0d4e587a0315996329f0d832791291916e51be9d1fefb73c3926fa" exitCode=0
Oct 30 04:12:20.269962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:20.269597    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz" event={"ID":"2ff56652-26ec-4dc5-918d-e20a420a307d","Type":"ContainerDied","Data":"7e79a9ca7c0d4e587a0315996329f0d832791291916e51be9d1fefb73c3926fa"}
Oct 30 04:12:21.427838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.427812    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:21.504526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.504488    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca\") pod \"2ff56652-26ec-4dc5-918d-e20a420a307d\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") "
Oct 30 04:12:21.504733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.504539    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h\" (UniqueName: \"kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h\") pod \"2ff56652-26ec-4dc5-918d-e20a420a307d\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") "
Oct 30 04:12:21.504733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.504573    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca\") pod \"2ff56652-26ec-4dc5-918d-e20a420a307d\" (UID: \"2ff56652-26ec-4dc5-918d-e20a420a307d\") "
Oct 30 04:12:21.504969 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.504932    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca" pod "2ff56652-26ec-4dc5-918d-e20a420a307d" (UID: "2ff56652-26ec-4dc5-918d-e20a420a307d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:21.506876 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.506850    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca" pod "2ff56652-26ec-4dc5-918d-e20a420a307d" (UID: "2ff56652-26ec-4dc5-918d-e20a420a307d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:21.506976 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.506879    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h" pod "2ff56652-26ec-4dc5-918d-e20a420a307d" (UID: "2ff56652-26ec-4dc5-918d-e20a420a307d"). InnerVolumeSpecName "kube-api-access-s5g7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:21.606382 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.606294    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:21.606382 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.606331    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5g7h\" (UniqueName: \"kubernetes.io/projected/2ff56652-26ec-4dc5-918d-e20a420a307d-kube-api-access-s5g7h\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:21.606382 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:21.606342    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff56652-26ec-4dc5-918d-e20a420a307d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:22.283005 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:22.282971    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"
Oct 30 04:12:22.283005 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:22.282991    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz" event={"ID":"2ff56652-26ec-4dc5-918d-e20a420a307d","Type":"ContainerDied","Data":"61a1849af1235c190a27938598d4c82cb3b536a0192dbd7234c88a26e1a992b8"}
Oct 30 04:12:22.283233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:22.283048    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61a1849af1235c190a27938598d4c82cb3b536a0192dbd7234c88a26e1a992b8"
Oct 30 04:12:30.025749 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:30.025711    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"]
Oct 30 04:12:30.035270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:30.035238    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6073175-2jbgq"]
Oct 30 04:12:31.697343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:31.697308    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963c2032-7eb9-4e3c-aa24-9dc979d80371" path="/var/lib/kubelet/pods/963c2032-7eb9-4e3c-aa24-9dc979d80371/volumes"
Oct 30 04:12:46.228989 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:12:46.228897    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:12:51.705647 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:51.705617    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"]
Oct 30 04:12:52.473193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.473134    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" podUID="04aa5563-67eb-4cbd-abe6-af26702f0252" containerName="discovery" containerID="cri-o://704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c" gracePeriod=30
Oct 30 04:12:52.739733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.739676    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 04:12:52.804603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.804558    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.804805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.804656    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.804805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.804778    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.804966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.804857    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.804966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.804905    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.805225    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.805299    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.805335    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs\") pod \"04aa5563-67eb-4cbd-abe6-af26702f0252\" (UID: \"04aa5563-67eb-4cbd-abe6-af26702f0252\") "
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.805761    4877 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-ca-configmap\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.807669    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.807699    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "kube-api-access-f94cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.807699    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.808045    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.808120    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:52.811454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.808538    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts" pod "04aa5563-67eb-4cbd-abe6-af26702f0252" (UID: "04aa5563-67eb-4cbd-abe6-af26702f0252"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:52.907015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.906971    4877 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-kubeconfig\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.907015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.907007    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.907015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.907021    4877 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-istio-csr-dns-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.907270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.907036    4877 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aa5563-67eb-4cbd-abe6-af26702f0252-local-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.907270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.907049    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f94cq\" (UniqueName: \"kubernetes.io/projected/04aa5563-67eb-4cbd-abe6-af26702f0252-kube-api-access-f94cq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.907270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:52.907063    4877 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04aa5563-67eb-4cbd-abe6-af26702f0252-cacerts\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:12:53.080187 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.080158    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:12:53.104961 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.104935    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:12:53.480675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.480583    4877 generic.go:356] "Generic (PLEG): container finished" podID="04aa5563-67eb-4cbd-abe6-af26702f0252" containerID="704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c" exitCode=0
Oct 30 04:12:53.480675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.480629    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"
Oct 30 04:12:53.480675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.480668    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" event={"ID":"04aa5563-67eb-4cbd-abe6-af26702f0252","Type":"ContainerDied","Data":"704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c"}
Oct 30 04:12:53.480954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.480718    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn" event={"ID":"04aa5563-67eb-4cbd-abe6-af26702f0252","Type":"ContainerDied","Data":"40f6be7413474125ca0896f309011366e1c7439f9113660d92272829ee84100e"}
Oct 30 04:12:53.480954 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.480745    4877 scope.go:122] "RemoveContainer" containerID="704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c"
Oct 30 04:12:53.493582 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.493560    4877 scope.go:122] "RemoveContainer" containerID="704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c"
Oct 30 04:12:53.493840 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:12:53.493819    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c\": container with ID starting with 704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c not found: ID does not exist" containerID="704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c"
Oct 30 04:12:53.493888 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.493850    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c"} err="failed to get container status \"704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c\": rpc error: code = NotFound desc = could not find container \"704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c\": container with ID starting with 704a7ed4af48fcb9a3fce427b9b5bf27fc0f6d05b7200f7ab47587fac410e72c not found: ID does not exist"
Oct 30 04:12:53.536579 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.536550    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"]
Oct 30 04:12:53.545757 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.545724    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgmcn"]
Oct 30 04:12:53.697109 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:53.697073    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04aa5563-67eb-4cbd-abe6-af26702f0252" path="/var/lib/kubelet/pods/04aa5563-67eb-4cbd-abe6-af26702f0252/volumes"
Oct 30 04:12:57.787583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:57.787552    4877 scope.go:122] "RemoveContainer" containerID="ead0d30745c6cf89b51c64fe73a207a333e27a599b57601fe0ccf058c48fa7ca"
Oct 30 04:12:57.797075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:57.797048    4877 scope.go:122] "RemoveContainer" containerID="720d11482213a245e883a0942202b4ed1be498d9d54b6f87569ac0411ddd55e6"
Oct 30 04:12:57.806172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:57.806146    4877 scope.go:122] "RemoveContainer" containerID="c977e3709945dd868c81df9a60d6bee2246692f64dd1311b4372bd690d2e6f91"
Oct 30 04:12:58.807154 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807119    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"]
Oct 30 04:12:58.807597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807581    4877 cpu_manager.go:414] "containerMap: removing container" podUID="2ff56652-26ec-4dc5-918d-e20a420a307d" containerName="revoke-keys"
Oct 30 04:12:58.807658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807598    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff56652-26ec-4dc5-918d-e20a420a307d" containerName="revoke-keys"
Oct 30 04:12:58.807658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807632    4877 cpu_manager.go:414] "containerMap: removing container" podUID="04aa5563-67eb-4cbd-abe6-af26702f0252" containerName="discovery"
Oct 30 04:12:58.807658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807638    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04aa5563-67eb-4cbd-abe6-af26702f0252" containerName="discovery"
Oct 30 04:12:58.807754 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807705    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="04aa5563-67eb-4cbd-abe6-af26702f0252" containerName="discovery"
Oct 30 04:12:58.807754 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.807717    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff56652-26ec-4dc5-918d-e20a420a307d" containerName="revoke-keys"
Oct 30 04:12:58.812741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.812722    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.835932 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.835897    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:12:58.836099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.835897    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:12:58.839826 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.839800    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"]
Oct 30 04:12:58.885796 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.885757    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8nt\" (UniqueName: \"kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.886004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.885826    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.886004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.885893    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.987342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.987296    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8nt\" (UniqueName: \"kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.987560 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.987355    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.987560 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.987498    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.988490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.988418    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:58.989857 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:58.989833    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:59.003984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:59.003955    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8nt\" (UniqueName: \"kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt\") pod \"maas-api-revoke-sub-f904bff9-xrcjq\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:59.126480 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:59.126382    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:12:59.428571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:59.428515    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"]
Oct 30 04:12:59.431478 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:12:59.431415    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3708a65b_caf3_4e2e_bbec_f36f1505c2db.slice/crio-571b0856b112da87757c461a1b03de7206c8af18f82d876e7363817f5ba047ef WatchSource:0}: Error finding container 571b0856b112da87757c461a1b03de7206c8af18f82d876e7363817f5ba047ef: Status 404 returned error can't find the container with id 571b0856b112da87757c461a1b03de7206c8af18f82d876e7363817f5ba047ef
Oct 30 04:12:59.542280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:12:59.542242    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq" event={"ID":"3708a65b-caf3-4e2e-bbec-f36f1505c2db","Type":"ContainerStarted","Data":"571b0856b112da87757c461a1b03de7206c8af18f82d876e7363817f5ba047ef"}
Oct 30 04:13:00.549730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:00.549639    4877 generic.go:356] "Generic (PLEG): container finished" podID="3708a65b-caf3-4e2e-bbec-f36f1505c2db" containerID="2efcf9b9a3aa2afbcb1b526c6445fb0dd069d84a9dc6fb046cd582a2eb0607b1" exitCode=0
Oct 30 04:13:00.549730 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:00.549702    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq" event={"ID":"3708a65b-caf3-4e2e-bbec-f36f1505c2db","Type":"ContainerDied","Data":"2efcf9b9a3aa2afbcb1b526c6445fb0dd069d84a9dc6fb046cd582a2eb0607b1"}
Oct 30 04:13:01.023760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.023724    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"]
Oct 30 04:13:01.033901 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.033873    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284a7d5b-b5hxv"]
Oct 30 04:13:01.698321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.698290    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec98d1d5-a261-472c-b639-5ce3002206c9" path="/var/lib/kubelet/pods/ec98d1d5-a261-472c-b639-5ce3002206c9/volumes"
Oct 30 04:13:01.700773 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.700751    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:13:01.813920 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.813824    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca\") pod \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") "
Oct 30 04:13:01.813920 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.813893    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca\") pod \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") "
Oct 30 04:13:01.814208 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.813979    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt\" (UniqueName: \"kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt\") pod \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\" (UID: \"3708a65b-caf3-4e2e-bbec-f36f1505c2db\") "
Oct 30 04:13:01.814282 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.814251    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca" pod "3708a65b-caf3-4e2e-bbec-f36f1505c2db" (UID: "3708a65b-caf3-4e2e-bbec-f36f1505c2db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:01.816105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.816079    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca" pod "3708a65b-caf3-4e2e-bbec-f36f1505c2db" (UID: "3708a65b-caf3-4e2e-bbec-f36f1505c2db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:01.816236 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.816120    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt" pod "3708a65b-caf3-4e2e-bbec-f36f1505c2db" (UID: "3708a65b-caf3-4e2e-bbec-f36f1505c2db"). InnerVolumeSpecName "kube-api-access-4g8nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:01.914939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.914895    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8nt\" (UniqueName: \"kubernetes.io/projected/3708a65b-caf3-4e2e-bbec-f36f1505c2db-kube-api-access-4g8nt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:01.914939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.914935    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:01.914939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:01.914945    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3708a65b-caf3-4e2e-bbec-f36f1505c2db-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:02.567664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:02.567635    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"
Oct 30 04:13:02.567861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:02.567669    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq" event={"ID":"3708a65b-caf3-4e2e-bbec-f36f1505c2db","Type":"ContainerDied","Data":"571b0856b112da87757c461a1b03de7206c8af18f82d876e7363817f5ba047ef"}
Oct 30 04:13:02.567861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:02.567724    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571b0856b112da87757c461a1b03de7206c8af18f82d876e7363817f5ba047ef"
Oct 30 04:13:16.029887 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:16.029841    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"]
Oct 30 04:13:16.039509 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:16.039478    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-311aefa5-s5n94"]
Oct 30 04:13:16.234767 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:13:16.234716    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:13:17.697117 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:17.697087    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7503d4-3753-4199-a664-94a6f6739421" path="/var/lib/kubelet/pods/2e7503d4-3753-4199-a664-94a6f6739421/volumes"
Oct 30 04:13:18.024180 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:18.024148    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"]
Oct 30 04:13:18.033429 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:18.033390    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f3adcc-x46g8"]
Oct 30 04:13:19.698934 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:19.698894    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230ebb35-03a1-4859-9ae1-595e063b8faa" path="/var/lib/kubelet/pods/230ebb35-03a1-4859-9ae1-595e063b8faa/volumes"
Oct 30 04:13:23.327104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.327065    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"]
Oct 30 04:13:23.327733 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.327710    4877 cpu_manager.go:414] "containerMap: removing container" podUID="3708a65b-caf3-4e2e-bbec-f36f1505c2db" containerName="revoke-keys"
Oct 30 04:13:23.327827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.327735    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3708a65b-caf3-4e2e-bbec-f36f1505c2db" containerName="revoke-keys"
Oct 30 04:13:23.327899 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.327853    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="3708a65b-caf3-4e2e-bbec-f36f1505c2db" containerName="revoke-keys"
Oct 30 04:13:23.331357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.331335    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.349382 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.349355    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"]
Oct 30 04:13:23.356262 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.356234    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:13:23.356262 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.356255    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:13:23.471881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.471849    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.472075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.471910    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9rf\" (UniqueName: \"kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.472075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.472006    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.573004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.572967    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9rf\" (UniqueName: \"kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.573180 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.573010    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.573180 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.573072    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.573789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.573765    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.575438 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.575410    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.590156 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.590105    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9rf\" (UniqueName: \"kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf\") pod \"maas-api-revoke-sub-f03629db-qllm7\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.645347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.645324    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:23.944741 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:23.944713    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"]
Oct 30 04:13:23.947277 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:13:23.947249    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c89ced_23ee_4e35_9dca_3ec6db57f477.slice/crio-5a276b422640c5c414035cfab0e40962f10732701f92362a70fa59c44c264775 WatchSource:0}: Error finding container 5a276b422640c5c414035cfab0e40962f10732701f92362a70fa59c44c264775: Status 404 returned error can't find the container with id 5a276b422640c5c414035cfab0e40962f10732701f92362a70fa59c44c264775
Oct 30 04:13:24.774135 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:24.774095    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7" event={"ID":"57c89ced-23ee-4e35-9dca-3ec6db57f477","Type":"ContainerStarted","Data":"62fdb1a3a009b96aab21e54cf96bd5191d92895d39928e4d39f3776d5a4b0a5b"}
Oct 30 04:13:24.774591 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:24.774176    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7" event={"ID":"57c89ced-23ee-4e35-9dca-3ec6db57f477","Type":"ContainerStarted","Data":"5a276b422640c5c414035cfab0e40962f10732701f92362a70fa59c44c264775"}
Oct 30 04:13:24.808687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:24.808607    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7" podStartSLOduration=1.808594772 podStartE2EDuration="1.808594772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:24.806638225 +0000 UTC m=+4233.235527662" watchObservedRunningTime="2026-10-30 04:13:24.808594772 +0000 UTC m=+4233.237484189"
Oct 30 04:13:25.782323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:25.782286    4877 generic.go:356] "Generic (PLEG): container finished" podID="57c89ced-23ee-4e35-9dca-3ec6db57f477" containerID="62fdb1a3a009b96aab21e54cf96bd5191d92895d39928e4d39f3776d5a4b0a5b" exitCode=0
Oct 30 04:13:25.782749 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:25.782375    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7" event={"ID":"57c89ced-23ee-4e35-9dca-3ec6db57f477","Type":"ContainerDied","Data":"62fdb1a3a009b96aab21e54cf96bd5191d92895d39928e4d39f3776d5a4b0a5b"}
Oct 30 04:13:26.934792 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:26.934767    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:27.109999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.109908    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf\" (UniqueName: \"kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf\") pod \"57c89ced-23ee-4e35-9dca-3ec6db57f477\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") "
Oct 30 04:13:27.110193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.110093    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca\") pod \"57c89ced-23ee-4e35-9dca-3ec6db57f477\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") "
Oct 30 04:13:27.110193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.110143    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca\") pod \"57c89ced-23ee-4e35-9dca-3ec6db57f477\" (UID: \"57c89ced-23ee-4e35-9dca-3ec6db57f477\") "
Oct 30 04:13:27.110526 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.110501    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca" pod "57c89ced-23ee-4e35-9dca-3ec6db57f477" (UID: "57c89ced-23ee-4e35-9dca-3ec6db57f477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:27.112227 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.112202    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca" pod "57c89ced-23ee-4e35-9dca-3ec6db57f477" (UID: "57c89ced-23ee-4e35-9dca-3ec6db57f477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:27.112328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.112308    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf" pod "57c89ced-23ee-4e35-9dca-3ec6db57f477" (UID: "57c89ced-23ee-4e35-9dca-3ec6db57f477"). InnerVolumeSpecName "kube-api-access-td9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:27.212099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.212054    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:27.212099 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.212097    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c89ced-23ee-4e35-9dca-3ec6db57f477-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:27.212451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.212113    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9rf\" (UniqueName: \"kubernetes.io/projected/57c89ced-23ee-4e35-9dca-3ec6db57f477-kube-api-access-td9rf\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:27.797573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.797542    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"
Oct 30 04:13:27.797771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.797569    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7" event={"ID":"57c89ced-23ee-4e35-9dca-3ec6db57f477","Type":"ContainerDied","Data":"5a276b422640c5c414035cfab0e40962f10732701f92362a70fa59c44c264775"}
Oct 30 04:13:27.797771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:27.797618    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a276b422640c5c414035cfab0e40962f10732701f92362a70fa59c44c264775"
Oct 30 04:13:29.028494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:29.028455    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"]
Oct 30 04:13:29.037870 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:29.037842    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708a6f4a-r4z8c"]
Oct 30 04:13:29.697693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:29.697651    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8438eaf6-ec90-433e-9097-383466506fed" path="/var/lib/kubelet/pods/8438eaf6-ec90-433e-9097-383466506fed/volumes"
Oct 30 04:13:39.441839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.441795    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"]
Oct 30 04:13:39.442300 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.442218    4877 cpu_manager.go:414] "containerMap: removing container" podUID="57c89ced-23ee-4e35-9dca-3ec6db57f477" containerName="revoke-keys"
Oct 30 04:13:39.442300 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.442229    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c89ced-23ee-4e35-9dca-3ec6db57f477" containerName="revoke-keys"
Oct 30 04:13:39.442378 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.442321    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c89ced-23ee-4e35-9dca-3ec6db57f477" containerName="revoke-keys"
Oct 30 04:13:39.445613 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.445594    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.457911 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.457890    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:13:39.463446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.463399    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.463535 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.463503    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtv8\" (UniqueName: \"kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.463623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.463610    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.467841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.467819    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:13:39.471306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.471283    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"]
Oct 30 04:13:39.564892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.564849    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.565116 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.564905    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.565116 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.564935    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtv8\" (UniqueName: \"kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.565591 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.565557    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.567264 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.567245    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.581798 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.581773    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtv8\" (UniqueName: \"kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8\") pod \"maas-api-revoke-sub-0165d7d2-6k66k\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:39.758846 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:39.758755    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:40.059013 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:40.058978    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"]
Oct 30 04:13:40.062571 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:13:40.062543    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d4beab_4e48_41c3_9141_673143d8cf71.slice/crio-3c64cdbf1be6c7bd80a507ec7c94e5202fae26b6eb05621309a0133fe94a96eb WatchSource:0}: Error finding container 3c64cdbf1be6c7bd80a507ec7c94e5202fae26b6eb05621309a0133fe94a96eb: Status 404 returned error can't find the container with id 3c64cdbf1be6c7bd80a507ec7c94e5202fae26b6eb05621309a0133fe94a96eb
Oct 30 04:13:40.890500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:40.890463    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k" event={"ID":"e7d4beab-4e48-41c3-9141-673143d8cf71","Type":"ContainerStarted","Data":"15a2bb2430f0f5e42a4e783545d41d87a74e9f4c08c2f69af62adc003f60d91b"}
Oct 30 04:13:40.890988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:40.890517    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k" event={"ID":"e7d4beab-4e48-41c3-9141-673143d8cf71","Type":"ContainerStarted","Data":"3c64cdbf1be6c7bd80a507ec7c94e5202fae26b6eb05621309a0133fe94a96eb"}
Oct 30 04:13:40.927555 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:40.927492    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k" podStartSLOduration=1.9274765839999999 podStartE2EDuration="1.927476584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:40.926165163 +0000 UTC m=+4249.355054581" watchObservedRunningTime="2026-10-30 04:13:40.927476584 +0000 UTC m=+4249.356366002"
Oct 30 04:13:41.897868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:41.897832    4877 generic.go:356] "Generic (PLEG): container finished" podID="e7d4beab-4e48-41c3-9141-673143d8cf71" containerID="15a2bb2430f0f5e42a4e783545d41d87a74e9f4c08c2f69af62adc003f60d91b" exitCode=0
Oct 30 04:13:41.898270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:41.897899    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k" event={"ID":"e7d4beab-4e48-41c3-9141-673143d8cf71","Type":"ContainerDied","Data":"15a2bb2430f0f5e42a4e783545d41d87a74e9f4c08c2f69af62adc003f60d91b"}
Oct 30 04:13:43.039874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.039849    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:43.107232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.107192    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca\") pod \"e7d4beab-4e48-41c3-9141-673143d8cf71\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") "
Oct 30 04:13:43.107415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.107287    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca\") pod \"e7d4beab-4e48-41c3-9141-673143d8cf71\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") "
Oct 30 04:13:43.107415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.107330    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8\" (UniqueName: \"kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8\") pod \"e7d4beab-4e48-41c3-9141-673143d8cf71\" (UID: \"e7d4beab-4e48-41c3-9141-673143d8cf71\") "
Oct 30 04:13:43.107600 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.107573    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca" pod "e7d4beab-4e48-41c3-9141-673143d8cf71" (UID: "e7d4beab-4e48-41c3-9141-673143d8cf71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:43.107818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.107791    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:43.110008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.109985    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8" pod "e7d4beab-4e48-41c3-9141-673143d8cf71" (UID: "e7d4beab-4e48-41c3-9141-673143d8cf71"). InnerVolumeSpecName "kube-api-access-bgtv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:43.110097 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.109989    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca" pod "e7d4beab-4e48-41c3-9141-673143d8cf71" (UID: "e7d4beab-4e48-41c3-9141-673143d8cf71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:43.208838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.208756    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d4beab-4e48-41c3-9141-673143d8cf71-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:43.208838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.208789    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgtv8\" (UniqueName: \"kubernetes.io/projected/e7d4beab-4e48-41c3-9141-673143d8cf71-kube-api-access-bgtv8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:43.911006 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.910973    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"
Oct 30 04:13:43.911214 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.911006    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k" event={"ID":"e7d4beab-4e48-41c3-9141-673143d8cf71","Type":"ContainerDied","Data":"3c64cdbf1be6c7bd80a507ec7c94e5202fae26b6eb05621309a0133fe94a96eb"}
Oct 30 04:13:43.911214 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:43.911054    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c64cdbf1be6c7bd80a507ec7c94e5202fae26b6eb05621309a0133fe94a96eb"
Oct 30 04:13:46.228549 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:13:46.228514    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:13:50.415937 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.415890    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"]
Oct 30 04:13:50.416662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.416639    4877 cpu_manager.go:414] "containerMap: removing container" podUID="e7d4beab-4e48-41c3-9141-673143d8cf71" containerName="revoke-keys"
Oct 30 04:13:50.416662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.416661    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d4beab-4e48-41c3-9141-673143d8cf71" containerName="revoke-keys"
Oct 30 04:13:50.416845 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.416793    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d4beab-4e48-41c3-9141-673143d8cf71" containerName="revoke-keys"
Oct 30 04:13:50.422776 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.422756    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.436609 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.436582    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"]
Oct 30 04:13:50.442820 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.442795    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:13:50.443552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.443531    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:13:50.480465 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.480405    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.480672 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.480642    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dp9w\" (UniqueName: \"kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.480738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.480694    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.581446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.581375    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.581636 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.581575    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dp9w\" (UniqueName: \"kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.581636 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.581611    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.582378 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.582347    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.583990 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.583967    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.598930 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.598902    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dp9w\" (UniqueName: \"kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w\") pod \"maas-api-revoke-sub-6b19338f-wjvfk\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:50.740642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:50.740561    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:51.044571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:51.044540    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"]
Oct 30 04:13:51.047454 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:13:51.047410    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdd374c_d218_484e_ad5d_2724aefca163.slice/crio-36f350c1206dd0b51c3a96abf67179e92a5bdf8999b295e0ac8248327659c370 WatchSource:0}: Error finding container 36f350c1206dd0b51c3a96abf67179e92a5bdf8999b295e0ac8248327659c370: Status 404 returned error can't find the container with id 36f350c1206dd0b51c3a96abf67179e92a5bdf8999b295e0ac8248327659c370
Oct 30 04:13:51.961115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:51.961074    4877 generic.go:356] "Generic (PLEG): container finished" podID="9bdd374c-d218-484e-ad5d-2724aefca163" containerID="348bdc1311f8a41c78c76d86c41913e4351924fa4b565f66d95d2de28c1545fa" exitCode=0
Oct 30 04:13:51.961497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:51.961120    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk" event={"ID":"9bdd374c-d218-484e-ad5d-2724aefca163","Type":"ContainerDied","Data":"348bdc1311f8a41c78c76d86c41913e4351924fa4b565f66d95d2de28c1545fa"}
Oct 30 04:13:51.961497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:51.961170    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk" event={"ID":"9bdd374c-d218-484e-ad5d-2724aefca163","Type":"ContainerStarted","Data":"36f350c1206dd0b51c3a96abf67179e92a5bdf8999b295e0ac8248327659c370"}
Oct 30 04:13:53.115951 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.115926    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:53.206836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.206800    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca\") pod \"9bdd374c-d218-484e-ad5d-2724aefca163\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") "
Oct 30 04:13:53.207024 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.206887    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca\") pod \"9bdd374c-d218-484e-ad5d-2724aefca163\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") "
Oct 30 04:13:53.207024 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.206932    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w\" (UniqueName: \"kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w\") pod \"9bdd374c-d218-484e-ad5d-2724aefca163\" (UID: \"9bdd374c-d218-484e-ad5d-2724aefca163\") "
Oct 30 04:13:53.207269 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.207239    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca" pod "9bdd374c-d218-484e-ad5d-2724aefca163" (UID: "9bdd374c-d218-484e-ad5d-2724aefca163"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:13:53.209128 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.209097    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w" pod "9bdd374c-d218-484e-ad5d-2724aefca163" (UID: "9bdd374c-d218-484e-ad5d-2724aefca163"). InnerVolumeSpecName "kube-api-access-8dp9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:13:53.209250 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.209136    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca" pod "9bdd374c-d218-484e-ad5d-2724aefca163" (UID: "9bdd374c-d218-484e-ad5d-2724aefca163"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:13:53.308204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.308175    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:53.308204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.308201    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bdd374c-d218-484e-ad5d-2724aefca163-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:53.308204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.308211    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dp9w\" (UniqueName: \"kubernetes.io/projected/9bdd374c-d218-484e-ad5d-2724aefca163-kube-api-access-8dp9w\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:13:53.974301 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.974226    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"
Oct 30 04:13:53.974488 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.974224    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk" event={"ID":"9bdd374c-d218-484e-ad5d-2724aefca163","Type":"ContainerDied","Data":"36f350c1206dd0b51c3a96abf67179e92a5bdf8999b295e0ac8248327659c370"}
Oct 30 04:13:53.974488 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:53.974362    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f350c1206dd0b51c3a96abf67179e92a5bdf8999b295e0ac8248327659c370"
Oct 30 04:13:57.845192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:57.845160    4877 scope.go:122] "RemoveContainer" containerID="bc13d6b7a3ab19b4ec8a96956bdce459260ee126b0befb2086beb50ba4de2e48"
Oct 30 04:13:57.854759 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:57.854736    4877 scope.go:122] "RemoveContainer" containerID="b2a86cae0ce5eaee96b32534e075685b287ec4d4f3e9c502715ad5e1d9028d38"
Oct 30 04:13:57.863451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:57.863415    4877 scope.go:122] "RemoveContainer" containerID="c8df4e6049c8aa743effaf635914dddfc376b62df76b367756a1aaf227cfccc6"
Oct 30 04:13:57.871577 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:13:57.871560    4877 scope.go:122] "RemoveContainer" containerID="7bdbca38976418cffa9a135af9e8f0b2a8d91589d09b64be7c9ed93a92d1b097"
Oct 30 04:14:03.339447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.336222    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"]
Oct 30 04:14:03.347537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.347071    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9bdd374c-d218-484e-ad5d-2724aefca163" containerName="revoke-keys"
Oct 30 04:14:03.347537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.347103    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdd374c-d218-484e-ad5d-2724aefca163" containerName="revoke-keys"
Oct 30 04:14:03.347537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.347244    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdd374c-d218-484e-ad5d-2724aefca163" containerName="revoke-keys"
Oct 30 04:14:03.360707 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.360679    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.365355 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.365323    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"]
Oct 30 04:14:03.385620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.385578    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-f2b786ee-kserve-self-signed-certs\""
Oct 30 04:14:03.460448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.453390    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.460448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.453456    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.460448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.453494    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.460448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.453519    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.460448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.453579    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bzx\" (UniqueName: \"kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.460448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.453621    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.561816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.561767    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.564539 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.563875    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.564777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.564748    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.565456 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.565219    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.565717 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.565698    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bzx\" (UniqueName: \"kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.566502 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.566473    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.566727 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.566517    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.566944 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.566925    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.567079 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.567050    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.567377 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.567351    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.568313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.568289    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.642295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.642186    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bzx\" (UniqueName: \"kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx\") pod \"e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.703657 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.703624    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:03.767841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.766254    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"]
Oct 30 04:14:03.773830 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.773791    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"]
Oct 30 04:14:03.773962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.773914    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:03.796470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.795811    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-f2b786ee-epp-sa-dockercfg-lsbjb\""
Oct 30 04:14:03.876104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.873869    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5kh\" (UniqueName: \"kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh\") pod \"e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:03.876104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.873930    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs\") pod \"e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:03.975029 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.974937    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5kh\" (UniqueName: \"kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh\") pod \"e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:03.975029 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.975001    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs\") pod \"e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:03.978249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.978213    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs\") pod \"e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:03.993732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:03.993651    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5kh\" (UniqueName: \"kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh\") pod \"e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") " pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:04.117630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:04.117601    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"]
Oct 30 04:14:04.120025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:04.120000    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:04.120389 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:14:04.120367    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fca6816_6a83_4e6e_8886_bce158d42abb.slice/crio-130d0d80cb661b1c9c4a0bd5ff3d7fac80b652dfc2dbc6235f305bd9ff729717 WatchSource:0}: Error finding container 130d0d80cb661b1c9c4a0bd5ff3d7fac80b652dfc2dbc6235f305bd9ff729717: Status 404 returned error can't find the container with id 130d0d80cb661b1c9c4a0bd5ff3d7fac80b652dfc2dbc6235f305bd9ff729717
Oct 30 04:14:04.419326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:04.419300    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"]
Oct 30 04:14:04.422275 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:14:04.422245    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0d9f20_79a5_46d9_a864_a33ba0a5bfbd.slice/crio-0182eb1c5112cbb2e54aa7451e7238c40e28069584e4d6bd400185b7cf01ef32 WatchSource:0}: Error finding container 0182eb1c5112cbb2e54aa7451e7238c40e28069584e4d6bd400185b7cf01ef32: Status 404 returned error can't find the container with id 0182eb1c5112cbb2e54aa7451e7238c40e28069584e4d6bd400185b7cf01ef32
Oct 30 04:14:04.425102 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:04.425082    4877 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:14:05.063478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:05.063443    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h" event={"ID":"6fca6816-6a83-4e6e-8886-bce158d42abb","Type":"ContainerStarted","Data":"4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30"}
Oct 30 04:14:05.063679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:05.063493    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h" event={"ID":"6fca6816-6a83-4e6e-8886-bce158d42abb","Type":"ContainerStarted","Data":"130d0d80cb661b1c9c4a0bd5ff3d7fac80b652dfc2dbc6235f305bd9ff729717"}
Oct 30 04:14:05.064895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:05.064870    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" event={"ID":"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd","Type":"ContainerStarted","Data":"0182eb1c5112cbb2e54aa7451e7238c40e28069584e4d6bd400185b7cf01ef32"}
Oct 30 04:14:05.100602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:05.100552    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h" podStartSLOduration=2.100540735 podStartE2EDuration="2.100540735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:05.097280265 +0000 UTC m=+4273.526169684" watchObservedRunningTime="2026-10-30 04:14:05.100540735 +0000 UTC m=+4273.529430153"
Oct 30 04:14:05.704806 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:05.704770    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:07.083534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:07.083490    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" event={"ID":"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd","Type":"ContainerStarted","Data":"58952b04a4270753fce85e6d86204f68731fd69dfa7959b3c509725a69e1d296"}
Oct 30 04:14:07.084115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:07.083575    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:07.117400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:07.117341    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podStartSLOduration=2.37280075 podStartE2EDuration="4.117321513s" totalImagesPullingTime="1.744520763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:07.116272789 +0000 UTC m=+4275.545162221" watchObservedRunningTime="2026-10-30 04:14:07.117321513 +0000 UTC m=+4275.546210937"
Oct 30 04:14:10.037517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:10.037477    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"]
Oct 30 04:14:10.046915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:10.046885    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51b37a0-rt6zs"]
Oct 30 04:14:11.697557 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:11.697517    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9933ea9c-eebb-45e7-93b4-4b56fc4dfd15" path="/var/lib/kubelet/pods/9933ea9c-eebb-45e7-93b4-4b56fc4dfd15/volumes"
Oct 30 04:14:13.704679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:13.704635    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:13.717958 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:13.717927    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:14.148870 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:14.148842    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:14:16.228233 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:14:16.228198    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:14:20.027125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:20.027084    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"]
Oct 30 04:14:20.036534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:20.036502    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271dc16c-2qqpd"]
Oct 30 04:14:21.697278 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:21.697243    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c572573-47ef-4b27-9e13-c1a561d30b9f" path="/var/lib/kubelet/pods/1c572573-47ef-4b27-9e13-c1a561d30b9f/volumes"
Oct 30 04:14:39.107147 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:39.107120    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:14:46.228942 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:14:46.228898    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:14:57.906348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:57.906312    4877 scope.go:122] "RemoveContainer" containerID="6dd134be51dbc5ab46866fa856a6e5498e61a47e751381593b46db4b812bb2c3"
Oct 30 04:14:57.915574 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:57.915550    4877 scope.go:122] "RemoveContainer" containerID="3e5ab2f219017bdf0a3b1d195e5231f51b48ce3b9fdedabe028d3291cf88eb4c"
Oct 30 04:14:57.923923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:14:57.923904    4877 scope.go:122] "RemoveContainer" containerID="8d10a3919fb311f600bba98b74cc218d5d2c23fbba8352c5435d8b762013a1ff"
Oct 30 04:15:05.535041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:05.535005    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"]
Oct 30 04:15:05.542605 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:05.542567    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"]
Oct 30 04:15:08.622090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.622045    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"]
Oct 30 04:15:08.625972 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.625947    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.638367 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.638345    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:15:08.648302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.648273    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:15:08.651482 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.651456    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"]
Oct 30 04:15:08.732684 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.732641    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.732875 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.732703    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.732875 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.732804    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwlh\" (UniqueName: \"kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.833401 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.833359    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.833634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.833499    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.833712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.833691    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwlh\" (UniqueName: \"kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.834242 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.834217    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.836051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.836027    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.850966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.850941    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwlh\" (UniqueName: \"kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh\") pod \"maas-api-revoke-sub-68394a88-v22hz\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:08.939823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:08.939752    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:09.107008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:09.106973    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:15:09.238393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:09.238284    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"]
Oct 30 04:15:09.241657 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:15:09.241626    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75710f4a_5000_4755_9a24_381fbdfd0188.slice/crio-a0ace49bf6f055b95c8553a6411822bb6059f40270f0354d08575514a0baeb79 WatchSource:0}: Error finding container a0ace49bf6f055b95c8553a6411822bb6059f40270f0354d08575514a0baeb79: Status 404 returned error can't find the container with id a0ace49bf6f055b95c8553a6411822bb6059f40270f0354d08575514a0baeb79
Oct 30 04:15:09.500941 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:09.500838    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz" event={"ID":"75710f4a-5000-4755-9a24-381fbdfd0188","Type":"ContainerStarted","Data":"a0ace49bf6f055b95c8553a6411822bb6059f40270f0354d08575514a0baeb79"}
Oct 30 04:15:10.543093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:10.543054    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz" event={"ID":"75710f4a-5000-4755-9a24-381fbdfd0188","Type":"ContainerStarted","Data":"cab033d98319b82872559fab8b24d2794e66aa327935fb932d13026ff8de8110"}
Oct 30 04:15:10.608451 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:10.604982    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz" podStartSLOduration=2.6049525669999998 podStartE2EDuration="2.604952567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:10.586964155 +0000 UTC m=+4339.015853597" watchObservedRunningTime="2026-10-30 04:15:10.604952567 +0000 UTC m=+4339.033841982"
Oct 30 04:15:11.559372 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:11.559284    4877 generic.go:356] "Generic (PLEG): container finished" podID="75710f4a-5000-4755-9a24-381fbdfd0188" containerID="cab033d98319b82872559fab8b24d2794e66aa327935fb932d13026ff8de8110" exitCode=0
Oct 30 04:15:11.559889 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:11.559332    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz" event={"ID":"75710f4a-5000-4755-9a24-381fbdfd0188","Type":"ContainerDied","Data":"cab033d98319b82872559fab8b24d2794e66aa327935fb932d13026ff8de8110"}
Oct 30 04:15:11.780447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:11.779190    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:15:11.780447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:11.779260    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:15:12.717902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.717872    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:12.784889 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.784847    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca\") pod \"75710f4a-5000-4755-9a24-381fbdfd0188\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") "
Oct 30 04:15:12.785109 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.784957    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh\" (UniqueName: \"kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh\") pod \"75710f4a-5000-4755-9a24-381fbdfd0188\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") "
Oct 30 04:15:12.785109 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.785057    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca\") pod \"75710f4a-5000-4755-9a24-381fbdfd0188\" (UID: \"75710f4a-5000-4755-9a24-381fbdfd0188\") "
Oct 30 04:15:12.785323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.785286    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca" pod "75710f4a-5000-4755-9a24-381fbdfd0188" (UID: "75710f4a-5000-4755-9a24-381fbdfd0188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:12.785473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.785373    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:12.787248 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.787224    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh" pod "75710f4a-5000-4755-9a24-381fbdfd0188" (UID: "75710f4a-5000-4755-9a24-381fbdfd0188"). InnerVolumeSpecName "kube-api-access-6gwlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:12.787625 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.787605    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca" pod "75710f4a-5000-4755-9a24-381fbdfd0188" (UID: "75710f4a-5000-4755-9a24-381fbdfd0188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:12.886068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.885986    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwlh\" (UniqueName: \"kubernetes.io/projected/75710f4a-5000-4755-9a24-381fbdfd0188-kube-api-access-6gwlh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:12.886068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:12.886018    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75710f4a-5000-4755-9a24-381fbdfd0188-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:13.574310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:13.574274    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"
Oct 30 04:15:13.574529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:13.574273    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz" event={"ID":"75710f4a-5000-4755-9a24-381fbdfd0188","Type":"ContainerDied","Data":"a0ace49bf6f055b95c8553a6411822bb6059f40270f0354d08575514a0baeb79"}
Oct 30 04:15:13.574529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:13.574448    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ace49bf6f055b95c8553a6411822bb6059f40270f0354d08575514a0baeb79"
Oct 30 04:15:16.230087 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:15:16.230047    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:15:19.120664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:19.120177    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:15:20.557105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.557058    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h" podUID="6fca6816-6a83-4e6e-8886-bce158d42abb" containerName="main" containerID="cri-o://4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30" gracePeriod=45
Oct 30 04:15:20.570071 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.570028    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main" containerID="cri-o://58952b04a4270753fce85e6d86204f68731fd69dfa7959b3c509725a69e1d296" gracePeriod=45
Oct 30 04:15:20.819892 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.819835    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:15:20.884776 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.884739    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir\") pod \"6fca6816-6a83-4e6e-8886-bce158d42abb\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") "
Oct 30 04:15:20.884971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.884814    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home\") pod \"6fca6816-6a83-4e6e-8886-bce158d42abb\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") "
Oct 30 04:15:20.884971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.884862    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs\") pod \"6fca6816-6a83-4e6e-8886-bce158d42abb\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") "
Oct 30 04:15:20.884971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.884901    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm\") pod \"6fca6816-6a83-4e6e-8886-bce158d42abb\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") "
Oct 30 04:15:20.884971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.884951    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache\") pod \"6fca6816-6a83-4e6e-8886-bce158d42abb\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") "
Oct 30 04:15:20.885215 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885013    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir" pod "6fca6816-6a83-4e6e-8886-bce158d42abb" (UID: "6fca6816-6a83-4e6e-8886-bce158d42abb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:20.885215 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885050    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx\" (UniqueName: \"kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx\") pod \"6fca6816-6a83-4e6e-8886-bce158d42abb\" (UID: \"6fca6816-6a83-4e6e-8886-bce158d42abb\") "
Oct 30 04:15:20.885215 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885129    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home" pod "6fca6816-6a83-4e6e-8886-bce158d42abb" (UID: "6fca6816-6a83-4e6e-8886-bce158d42abb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:20.885364 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885273    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache" pod "6fca6816-6a83-4e6e-8886-bce158d42abb" (UID: "6fca6816-6a83-4e6e-8886-bce158d42abb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:20.885473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885456    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.885544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885475    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.885544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.885483    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.887304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.887282    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx" pod "6fca6816-6a83-4e6e-8886-bce158d42abb" (UID: "6fca6816-6a83-4e6e-8886-bce158d42abb"). InnerVolumeSpecName "kube-api-access-s2bzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:20.887495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.887469    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm" pod "6fca6816-6a83-4e6e-8886-bce158d42abb" (UID: "6fca6816-6a83-4e6e-8886-bce158d42abb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:20.887648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.887633    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs" pod "6fca6816-6a83-4e6e-8886-bce158d42abb" (UID: "6fca6816-6a83-4e6e-8886-bce158d42abb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:20.986041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.986006    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bzx\" (UniqueName: \"kubernetes.io/projected/6fca6816-6a83-4e6e-8886-bce158d42abb-kube-api-access-s2bzx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.986041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.986036    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fca6816-6a83-4e6e-8886-bce158d42abb-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:20.986041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:20.986045    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fca6816-6a83-4e6e-8886-bce158d42abb-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:21.638600 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.638561    4877 generic.go:356] "Generic (PLEG): container finished" podID="6fca6816-6a83-4e6e-8886-bce158d42abb" containerID="4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30" exitCode=0
Oct 30 04:15:21.639118 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.638652    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"
Oct 30 04:15:21.639118 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.638659    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h" event={"ID":"6fca6816-6a83-4e6e-8886-bce158d42abb","Type":"ContainerDied","Data":"4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30"}
Oct 30 04:15:21.639118 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.638710    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h" event={"ID":"6fca6816-6a83-4e6e-8886-bce158d42abb","Type":"ContainerDied","Data":"130d0d80cb661b1c9c4a0bd5ff3d7fac80b652dfc2dbc6235f305bd9ff729717"}
Oct 30 04:15:21.639118 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.638732    4877 scope.go:122] "RemoveContainer" containerID="4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30"
Oct 30 04:15:21.656617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.653594    4877 scope.go:122] "RemoveContainer" containerID="4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30"
Oct 30 04:15:21.656710 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:15:21.656671    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30\": container with ID starting with 4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30 not found: ID does not exist" containerID="4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30"
Oct 30 04:15:21.656780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.656700    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30"} err="failed to get container status \"4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30\": rpc error: code = NotFound desc = could not find container \"4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30\": container with ID starting with 4d4ab02765624f3c5292f0bc513bfbdd4d77eb4e0a8d5ea06afd6886f1a52d30 not found: ID does not exist"
Oct 30 04:15:21.683793 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.683760    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"]
Oct 30 04:15:21.699028 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:21.698997    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-5458795b48-gv59h"]
Oct 30 04:15:23.697867 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:23.697824    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fca6816-6a83-4e6e-8886-bce158d42abb" path="/var/lib/kubelet/pods/6fca6816-6a83-4e6e-8886-bce158d42abb/volumes"
Oct 30 04:15:29.107589 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:29.107550    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:15:29.108111 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:29.107670    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:15:39.107400 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:39.107313    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:15:42.031960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:42.031916    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"]
Oct 30 04:15:42.041975 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:42.041937    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9eddf8-lrhjz"]
Oct 30 04:15:43.697306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:43.697268    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bce459d-bbe2-4433-ab19-bd65c9fbec8a" path="/var/lib/kubelet/pods/9bce459d-bbe2-4433-ab19-bd65c9fbec8a/volumes"
Oct 30 04:15:46.228732 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:15:46.228693    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:15:49.106878 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:49.106839    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:15:50.826662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.826632    4877 generic.go:356] "Generic (PLEG): container finished" podID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerID="58952b04a4270753fce85e6d86204f68731fd69dfa7959b3c509725a69e1d296" exitCode=0
Oct 30 04:15:50.827050 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.826706    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" event={"ID":"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd","Type":"ContainerDied","Data":"58952b04a4270753fce85e6d86204f68731fd69dfa7959b3c509725a69e1d296"}
Oct 30 04:15:50.827050 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.826775    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb" event={"ID":"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd","Type":"ContainerDied","Data":"0182eb1c5112cbb2e54aa7451e7238c40e28069584e4d6bd400185b7cf01ef32"}
Oct 30 04:15:50.827050 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.826806    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0182eb1c5112cbb2e54aa7451e7238c40e28069584e4d6bd400185b7cf01ef32"
Oct 30 04:15:50.839078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.839018    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:15:50.883358 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.883331    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh\" (UniqueName: \"kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh\") pod \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") "
Oct 30 04:15:50.883566 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.883466    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs\") pod \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\" (UID: \"fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd\") "
Oct 30 04:15:50.885602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.885569    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs" pod "fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" (UID: "fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:50.885713 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.885639    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh" pod "fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" (UID: "fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd"). InnerVolumeSpecName "kube-api-access-dq5kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:50.985304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.985263    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:50.985304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:50.985297    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5kh\" (UniqueName: \"kubernetes.io/projected/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd-kube-api-access-dq5kh\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:15:51.832999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:51.832967    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"
Oct 30 04:15:51.876816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:51.876775    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"]
Oct 30 04:15:51.886252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:51.886223    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-f2b786ee-kserve-router-scheduler-5cbcc9f68c-tblfb"]
Oct 30 04:15:53.025745 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:53.025705    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"]
Oct 30 04:15:53.035032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:53.035001    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b487de-c9lc7"]
Oct 30 04:15:53.696514 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:53.696480    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00db8e5-5be4-40cc-9cdb-ea14525dffd2" path="/var/lib/kubelet/pods/a00db8e5-5be4-40cc-9cdb-ea14525dffd2/volumes"
Oct 30 04:15:53.696881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:53.696868    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" path="/var/lib/kubelet/pods/fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd/volumes"
Oct 30 04:15:57.959068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:57.959033    4877 scope.go:122] "RemoveContainer" containerID="16eff3a2a7e167f96b59c51993505181258cdf7b87d379764079dfc4bd0bf8c8"
Oct 30 04:15:57.968755 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:15:57.968732    4877 scope.go:122] "RemoveContainer" containerID="41017cb05f335da0cf1a18b697f80640f40b5dc2945b4f69b978daa4a446d6c6"
Oct 30 04:16:12.201695 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.201649    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"]
Oct 30 04:16:12.202346 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202319    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6fca6816-6a83-4e6e-8886-bce158d42abb" containerName="main"
Oct 30 04:16:12.202346 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202340    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fca6816-6a83-4e6e-8886-bce158d42abb" containerName="main"
Oct 30 04:16:12.202534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202355    4877 cpu_manager.go:414] "containerMap: removing container" podUID="75710f4a-5000-4755-9a24-381fbdfd0188" containerName="revoke-keys"
Oct 30 04:16:12.202534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202363    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75710f4a-5000-4755-9a24-381fbdfd0188" containerName="revoke-keys"
Oct 30 04:16:12.202534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202385    4877 cpu_manager.go:414] "containerMap: removing container" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main"
Oct 30 04:16:12.202534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202393    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main"
Oct 30 04:16:12.202760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202558    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fca6816-6a83-4e6e-8886-bce158d42abb" containerName="main"
Oct 30 04:16:12.202760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202571    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="75710f4a-5000-4755-9a24-381fbdfd0188" containerName="revoke-keys"
Oct 30 04:16:12.202760 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.202583    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0d9f20-79a5-46d9-a864-a33ba0a5bfbd" containerName="main"
Oct 30 04:16:12.205979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.205956    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.220455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.220435    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:16:12.223856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.223822    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"]
Oct 30 04:16:12.230200 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.230171    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:16:12.405771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.405738    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.405949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.405862    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmk8\" (UniqueName: \"kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.405949 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.405934    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.506787 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.506711    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmk8\" (UniqueName: \"kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.506787 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.506751    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.506787 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.506780    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.507408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.507379    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.509114 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.509087    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.523219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.523195    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmk8\" (UniqueName: \"kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8\") pod \"maas-api-revoke-sub-e251bc8d-wpl2h\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:12.819231 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:12.819199    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:13.116973 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:13.116933    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"]
Oct 30 04:16:13.119575 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:16:13.119537    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f3d084_ef78_4333_84eb_01e36687b2cb.slice/crio-1cc06f543b0a54e1dee74e102e56ec6550d513a11cc0aa8e4ceab2e09083d2ce WatchSource:0}: Error finding container 1cc06f543b0a54e1dee74e102e56ec6550d513a11cc0aa8e4ceab2e09083d2ce: Status 404 returned error can't find the container with id 1cc06f543b0a54e1dee74e102e56ec6550d513a11cc0aa8e4ceab2e09083d2ce
Oct 30 04:16:13.975579 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:13.975535    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h" event={"ID":"c1f3d084-ef78-4333-84eb-01e36687b2cb","Type":"ContainerStarted","Data":"7f70d681f344460a8efdd42593e2ee0ba44ff9b54cf134c2cbbe919036c028ba"}
Oct 30 04:16:13.976263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:13.976218    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h" event={"ID":"c1f3d084-ef78-4333-84eb-01e36687b2cb","Type":"ContainerStarted","Data":"1cc06f543b0a54e1dee74e102e56ec6550d513a11cc0aa8e4ceab2e09083d2ce"}
Oct 30 04:16:14.014523 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:14.014465    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h" podStartSLOduration=2.014453026 podStartE2EDuration="2.014453026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:14.011386768 +0000 UTC m=+4402.440276186" watchObservedRunningTime="2026-10-30 04:16:14.014453026 +0000 UTC m=+4402.443342440"
Oct 30 04:16:14.982338 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:14.982301    4877 generic.go:356] "Generic (PLEG): container finished" podID="c1f3d084-ef78-4333-84eb-01e36687b2cb" containerID="7f70d681f344460a8efdd42593e2ee0ba44ff9b54cf134c2cbbe919036c028ba" exitCode=0
Oct 30 04:16:14.982778 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:14.982388    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h" event={"ID":"c1f3d084-ef78-4333-84eb-01e36687b2cb","Type":"ContainerDied","Data":"7f70d681f344460a8efdd42593e2ee0ba44ff9b54cf134c2cbbe919036c028ba"}
Oct 30 04:16:16.145833 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.145805    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:16.236157 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:16:16.236113    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:16:16.239612 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.239551    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca\") pod \"c1f3d084-ef78-4333-84eb-01e36687b2cb\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") "
Oct 30 04:16:16.239727 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.239619    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8\" (UniqueName: \"kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8\") pod \"c1f3d084-ef78-4333-84eb-01e36687b2cb\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") "
Oct 30 04:16:16.239727 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.239663    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca\") pod \"c1f3d084-ef78-4333-84eb-01e36687b2cb\" (UID: \"c1f3d084-ef78-4333-84eb-01e36687b2cb\") "
Oct 30 04:16:16.239973 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.239950    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca" pod "c1f3d084-ef78-4333-84eb-01e36687b2cb" (UID: "c1f3d084-ef78-4333-84eb-01e36687b2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:16.241704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.241678    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca" pod "c1f3d084-ef78-4333-84eb-01e36687b2cb" (UID: "c1f3d084-ef78-4333-84eb-01e36687b2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.241791 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.241707    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8" pod "c1f3d084-ef78-4333-84eb-01e36687b2cb" (UID: "c1f3d084-ef78-4333-84eb-01e36687b2cb"). InnerVolumeSpecName "kube-api-access-jvmk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:16.341270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.341226    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmk8\" (UniqueName: \"kubernetes.io/projected/c1f3d084-ef78-4333-84eb-01e36687b2cb-kube-api-access-jvmk8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.341270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.341264    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.341270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.341278    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d084-ef78-4333-84eb-01e36687b2cb-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.997005 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.996973    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"
Oct 30 04:16:16.997186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.996970    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h" event={"ID":"c1f3d084-ef78-4333-84eb-01e36687b2cb","Type":"ContainerDied","Data":"1cc06f543b0a54e1dee74e102e56ec6550d513a11cc0aa8e4ceab2e09083d2ce"}
Oct 30 04:16:16.997186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:16.997109    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc06f543b0a54e1dee74e102e56ec6550d513a11cc0aa8e4ceab2e09083d2ce"
Oct 30 04:16:42.085275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.085237    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"]
Oct 30 04:16:42.085950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.085797    4877 cpu_manager.go:414] "containerMap: removing container" podUID="c1f3d084-ef78-4333-84eb-01e36687b2cb" containerName="revoke-keys"
Oct 30 04:16:42.085950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.085812    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f3d084-ef78-4333-84eb-01e36687b2cb" containerName="revoke-keys"
Oct 30 04:16:42.085950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.085874    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f3d084-ef78-4333-84eb-01e36687b2cb" containerName="revoke-keys"
Oct 30 04:16:42.089584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.089560    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.119602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.119501    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d820fa78-kserve-self-signed-certs\""
Oct 30 04:16:42.123219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.123188    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"]
Oct 30 04:16:42.196811 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.196777    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lwt\" (UniqueName: \"kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.196811 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.196813    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.197049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.196832    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.197049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.196982    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.197122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.197052    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.197173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.197152    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298309 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298267    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298360    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lwt\" (UniqueName: \"kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298382    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298400    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298474    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298606 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298537    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298820    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298848    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.298977 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.298927    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.300828 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.300802    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.301089 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.301070    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.314909 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.314884    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lwt\" (UniqueName: \"kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt\") pod \"e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") " pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.406054 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.405973    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:42.703201 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:42.703171    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"]
Oct 30 04:16:42.704487 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:16:42.704457    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc253802_1ef6_4aa9_bf45_e0b9ea10e9ef.slice/crio-fed1f70189475391354187c907f520927609c59d778bbe95957eebcaf20f914b WatchSource:0}: Error finding container fed1f70189475391354187c907f520927609c59d778bbe95957eebcaf20f914b: Status 404 returned error can't find the container with id fed1f70189475391354187c907f520927609c59d778bbe95957eebcaf20f914b
Oct 30 04:16:43.160167 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:43.160121    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc" event={"ID":"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef","Type":"ContainerStarted","Data":"a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b"}
Oct 30 04:16:43.160598 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:43.160183    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc" event={"ID":"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef","Type":"ContainerStarted","Data":"fed1f70189475391354187c907f520927609c59d778bbe95957eebcaf20f914b"}
Oct 30 04:16:43.195443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:43.195367    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc" podStartSLOduration=1.1953544520000001 podStartE2EDuration="1.195354452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:43.194085404 +0000 UTC m=+4431.622974821" watchObservedRunningTime="2026-10-30 04:16:43.195354452 +0000 UTC m=+4431.624243869"
Oct 30 04:16:43.406253 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:43.406220    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:46.271308 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:16:46.271253    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:16:52.406990 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:52.406950    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:52.420243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:52.420219    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:53.275601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:53.275570    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:16:58.003209 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:58.003173    4877 scope.go:122] "RemoveContainer" containerID="89404ed884f62c3e5f40d0f01b615ad2f5b6d386cb8097c0e8b55a6650172f88"
Oct 30 04:16:58.012343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:16:58.012324    4877 scope.go:122] "RemoveContainer" containerID="dbd1c816a9e428c8639299cf0fa7a87d305e4922bf61f12dd8eee0e17df525ee"
Oct 30 04:17:15.540330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.540291    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"]
Oct 30 04:17:15.543924 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.543906    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.556221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.556194    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:17:15.566309 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.566286    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:17:15.569416 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.569394    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"]
Oct 30 04:17:15.672321 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.672292    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.672531 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.672330    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.672531 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.672467    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85ch\" (UniqueName: \"kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.773444 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.773390    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85ch\" (UniqueName: \"kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.773643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.773519    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.773643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.773548    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.774287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.774261    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.775964 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.775940    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.790003 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.789984    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85ch\" (UniqueName: \"kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch\") pod \"maas-api-revoke-sub-452b6d7b-dlhgv\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:15.856631 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:15.856552    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:16.154990 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:16.154959    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"]
Oct 30 04:17:16.156736 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:17:16.156711    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e496f11_ad9b_450e_ae36_e5ea8f1c32e1.slice/crio-a4b5564219e23af552144c4f684ac3cecc00190ae0250282cec78f41af6e7756 WatchSource:0}: Error finding container a4b5564219e23af552144c4f684ac3cecc00190ae0250282cec78f41af6e7756: Status 404 returned error can't find the container with id a4b5564219e23af552144c4f684ac3cecc00190ae0250282cec78f41af6e7756
Oct 30 04:17:16.229456 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:17:16.229403    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:17:16.452369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:16.452272    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv" event={"ID":"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1","Type":"ContainerStarted","Data":"a4b5564219e23af552144c4f684ac3cecc00190ae0250282cec78f41af6e7756"}
Oct 30 04:17:17.461592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:17.461543    4877 generic.go:356] "Generic (PLEG): container finished" podID="0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" containerID="fc2c04081ec0d6e3cdb2a96859f2642b18abafc1b2f1c340bb2c2544771708fc" exitCode=0
Oct 30 04:17:17.462129 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:17.461621    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv" event={"ID":"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1","Type":"ContainerDied","Data":"fc2c04081ec0d6e3cdb2a96859f2642b18abafc1b2f1c340bb2c2544771708fc"}
Oct 30 04:17:18.864199 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.864168    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:18.959705 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.959666    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca\") pod \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") "
Oct 30 04:17:18.962416 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.962378    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca" pod "0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" (UID: "0e496f11-ad9b-450e-ae36-e5ea8f1c32e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:18.962767 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.962743    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca" pod "0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" (UID: "0e496f11-ad9b-450e-ae36-e5ea8f1c32e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:18.962848 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.960057    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca\") pod \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") "
Oct 30 04:17:18.962848 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.962826    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch\" (UniqueName: \"kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch\") pod \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\" (UID: \"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1\") "
Oct 30 04:17:18.963224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.963204    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.963310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.963227    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:18.969123 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:18.969103    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch" pod "0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" (UID: "0e496f11-ad9b-450e-ae36-e5ea8f1c32e1"). InnerVolumeSpecName "kube-api-access-t85ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:19.064098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:19.064005    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t85ch\" (UniqueName: \"kubernetes.io/projected/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1-kube-api-access-t85ch\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:19.479402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:19.479309    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv" event={"ID":"0e496f11-ad9b-450e-ae36-e5ea8f1c32e1","Type":"ContainerDied","Data":"a4b5564219e23af552144c4f684ac3cecc00190ae0250282cec78f41af6e7756"}
Oct 30 04:17:19.479712 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:19.479688    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b5564219e23af552144c4f684ac3cecc00190ae0250282cec78f41af6e7756"
Oct 30 04:17:19.479908 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:19.479893    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"
Oct 30 04:17:22.026742 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:22.026708    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"]
Oct 30 04:17:22.034721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:22.034690    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xtdtz"]
Oct 30 04:17:23.698032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:23.697995    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff56652-26ec-4dc5-918d-e20a420a307d" path="/var/lib/kubelet/pods/2ff56652-26ec-4dc5-918d-e20a420a307d/volumes"
Oct 30 04:17:26.709646 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:26.709601    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"]
Oct 30 04:17:32.646556 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.646520    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"]
Oct 30 04:17:32.646962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.646933    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" containerName="revoke-keys"
Oct 30 04:17:32.646962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.646945    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" containerName="revoke-keys"
Oct 30 04:17:32.647043 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.647034    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" containerName="revoke-keys"
Oct 30 04:17:32.650236 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.650220    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.665748 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.665727    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:17:32.675869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.675845    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:17:32.680776 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.680733    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"]
Oct 30 04:17:32.760166 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.760127    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.760354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.760238    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxvw\" (UniqueName: \"kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.760354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.760272    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.861273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.861227    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.861273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.861288    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxvw\" (UniqueName: \"kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.861626 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.861322    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.862165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.862137    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.863853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.863829    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.878621 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.878589    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxvw\" (UniqueName: \"kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw\") pod \"maas-api-revoke-sub-3e31a506-bxtgj\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:32.962940 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:32.962857    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:33.264305 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:33.264276    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"]
Oct 30 04:17:33.266926 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:17:33.266888    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404e6fbd_d2a6_4ed1_93e6_a3907eaa7f57.slice/crio-d6a6144b70215a846b2f022d4ca0e90bf89849ffc3d08d11930bef58fd0ad75d WatchSource:0}: Error finding container d6a6144b70215a846b2f022d4ca0e90bf89849ffc3d08d11930bef58fd0ad75d: Status 404 returned error can't find the container with id d6a6144b70215a846b2f022d4ca0e90bf89849ffc3d08d11930bef58fd0ad75d
Oct 30 04:17:33.582096 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:33.582055    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj" event={"ID":"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57","Type":"ContainerStarted","Data":"d6a6144b70215a846b2f022d4ca0e90bf89849ffc3d08d11930bef58fd0ad75d"}
Oct 30 04:17:34.644515 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:34.644049    4877 generic.go:356] "Generic (PLEG): container finished" podID="404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" containerID="88833fde12df2353fc5c487263233a046e85433921fa703c140c6e99d36a25ec" exitCode=0
Oct 30 04:17:34.644515 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:34.644241    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj" event={"ID":"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57","Type":"ContainerDied","Data":"88833fde12df2353fc5c487263233a046e85433921fa703c140c6e99d36a25ec"}
Oct 30 04:17:35.880131 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:35.880076    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:17:35.880822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:35.880786    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:17:35.971931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:35.971904    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:36.053206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.053169    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw\" (UniqueName: \"kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw\") pod \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") "
Oct 30 04:17:36.053377 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.053292    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca\") pod \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") "
Oct 30 04:17:36.053377 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.053357    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca\") pod \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\" (UID: \"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57\") "
Oct 30 04:17:36.053710 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.053686    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca" pod "404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" (UID: "404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:36.055497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.055467    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw" pod "404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" (UID: "404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57"). InnerVolumeSpecName "kube-api-access-jzxvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:36.055604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.055501    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca" pod "404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" (UID: "404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:36.154206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.154175    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.154206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.154203    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.154206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.154213    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzxvw\" (UniqueName: \"kubernetes.io/projected/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57-kube-api-access-jzxvw\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:36.663010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.662964    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj" event={"ID":"404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57","Type":"ContainerDied","Data":"d6a6144b70215a846b2f022d4ca0e90bf89849ffc3d08d11930bef58fd0ad75d"}
Oct 30 04:17:36.663010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.662989    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"
Oct 30 04:17:36.663234 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:36.663026    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6a6144b70215a846b2f022d4ca0e90bf89849ffc3d08d11930bef58fd0ad75d"
Oct 30 04:17:41.730032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:41.729972    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc" podUID="bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" containerName="main" containerID="cri-o://a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b" gracePeriod=45
Oct 30 04:17:41.982106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:41.982045    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:17:42.115582 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.115549    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs\") pod \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") "
Oct 30 04:17:42.115779 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.115667    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm\") pod \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") "
Oct 30 04:17:42.115779 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.115693    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache\") pod \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") "
Oct 30 04:17:42.115779 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.115740    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir\") pod \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") "
Oct 30 04:17:42.115957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.115788    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home\") pod \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") "
Oct 30 04:17:42.115957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.115826    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt\" (UniqueName: \"kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt\") pod \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\" (UID: \"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef\") "
Oct 30 04:17:42.116240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.116195    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir" pod "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" (UID: "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:42.116240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.116218    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache" pod "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" (UID: "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:42.116240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.116231    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home" pod "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" (UID: "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:42.116551 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.116315    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:42.117903 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.117880    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm" pod "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" (UID: "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:42.118041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.118015    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs" pod "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" (UID: "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:42.118168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.118144    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt" pod "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" (UID: "bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef"). InnerVolumeSpecName "kube-api-access-k9lwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:42.217699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.217660    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:42.217699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.217693    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:42.217699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.217708    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:42.217948 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.217721    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lwt\" (UniqueName: \"kubernetes.io/projected/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-kube-api-access-k9lwt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:42.217948 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.217734    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:42.698129 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.698089    4877 generic.go:356] "Generic (PLEG): container finished" podID="bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" containerID="a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b" exitCode=0
Oct 30 04:17:42.698313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.698147    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"
Oct 30 04:17:42.698313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.698176    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc" event={"ID":"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef","Type":"ContainerDied","Data":"a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b"}
Oct 30 04:17:42.698313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.698228    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc" event={"ID":"bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef","Type":"ContainerDied","Data":"fed1f70189475391354187c907f520927609c59d778bbe95957eebcaf20f914b"}
Oct 30 04:17:42.698313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.698251    4877 scope.go:122] "RemoveContainer" containerID="a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b"
Oct 30 04:17:42.708300 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.708278    4877 scope.go:122] "RemoveContainer" containerID="a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b"
Oct 30 04:17:42.708641 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:17:42.708621    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b\": container with ID starting with a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b not found: ID does not exist" containerID="a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b"
Oct 30 04:17:42.708696 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.708652    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b"} err="failed to get container status \"a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b\": rpc error: code = NotFound desc = could not find container \"a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b\": container with ID starting with a2fabb3fa7d997fae93d93886c74b1dac69f3cc4206c822674bf589fbafc556b not found: ID does not exist"
Oct 30 04:17:42.741864 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.741836    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"]
Oct 30 04:17:42.751357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:42.751334    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d820fa78-kserve-5b8cfb96f-xtxrc"]
Oct 30 04:17:43.698080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:43.698046    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" path="/var/lib/kubelet/pods/bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef/volumes"
Oct 30 04:17:46.231542 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:17:46.231502    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:17:51.283383 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283345    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"]
Oct 30 04:17:51.284045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283781    4877 cpu_manager.go:414] "containerMap: removing container" podUID="404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" containerName="revoke-keys"
Oct 30 04:17:51.284045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283820    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" containerName="revoke-keys"
Oct 30 04:17:51.284045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283855    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" containerName="main"
Oct 30 04:17:51.284045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283861    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" containerName="main"
Oct 30 04:17:51.284045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283927    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc253802-1ef6-4aa9-bf45-e0b9ea10e9ef" containerName="main"
Oct 30 04:17:51.284045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.283937    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" containerName="revoke-keys"
Oct 30 04:17:51.288554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.288530    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.309824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.309799    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"]
Oct 30 04:17:51.314264 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.314240    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:17:51.314406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.314314    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:17:51.360150 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.360109    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.360354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.360268    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgs8\" (UniqueName: \"kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.360443 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.360365    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.461179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.461149    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.461357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.461215    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgs8\" (UniqueName: \"kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.461357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.461305    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.461902 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.461875    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.463829 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.463802    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.485943 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.485915    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgs8\" (UniqueName: \"kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8\") pod \"maas-api-revoke-sub-396e7f24-qmlmg\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.602349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.602267    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:51.898476 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:51.898405    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"]
Oct 30 04:17:51.900412 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:17:51.900383    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bd6584_7335_4f53_8753_6eec8a51d775.slice/crio-41d6294fb7e9d74a03075cb98f37b6abbf8b07277b8ac4745c676ba0376635cc WatchSource:0}: Error finding container 41d6294fb7e9d74a03075cb98f37b6abbf8b07277b8ac4745c676ba0376635cc: Status 404 returned error can't find the container with id 41d6294fb7e9d74a03075cb98f37b6abbf8b07277b8ac4745c676ba0376635cc
Oct 30 04:17:52.806379 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:52.806342    4877 generic.go:356] "Generic (PLEG): container finished" podID="a0bd6584-7335-4f53-8753-6eec8a51d775" containerID="9eef6905cd4ea2c5a918ee0b34a34ab9076f40e7dec695bd5f1646ad6ff698c6" exitCode=0
Oct 30 04:17:52.806836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:52.806437    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg" event={"ID":"a0bd6584-7335-4f53-8753-6eec8a51d775","Type":"ContainerDied","Data":"9eef6905cd4ea2c5a918ee0b34a34ab9076f40e7dec695bd5f1646ad6ff698c6"}
Oct 30 04:17:52.806836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:52.806490    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg" event={"ID":"a0bd6584-7335-4f53-8753-6eec8a51d775","Type":"ContainerStarted","Data":"41d6294fb7e9d74a03075cb98f37b6abbf8b07277b8ac4745c676ba0376635cc"}
Oct 30 04:17:53.138851 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.138813    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:17:53.161281 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.161255    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:17:53.954766 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.954740    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:53.981509 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.981470    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8\" (UniqueName: \"kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8\") pod \"a0bd6584-7335-4f53-8753-6eec8a51d775\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") "
Oct 30 04:17:53.981663 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.981646    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca\") pod \"a0bd6584-7335-4f53-8753-6eec8a51d775\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") "
Oct 30 04:17:53.981714 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.981692    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca\") pod \"a0bd6584-7335-4f53-8753-6eec8a51d775\" (UID: \"a0bd6584-7335-4f53-8753-6eec8a51d775\") "
Oct 30 04:17:53.982065 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.982039    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca" pod "a0bd6584-7335-4f53-8753-6eec8a51d775" (UID: "a0bd6584-7335-4f53-8753-6eec8a51d775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:53.983513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.983482    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8" pod "a0bd6584-7335-4f53-8753-6eec8a51d775" (UID: "a0bd6584-7335-4f53-8753-6eec8a51d775"). InnerVolumeSpecName "kube-api-access-8vgs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:53.983612 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:53.983584    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca" pod "a0bd6584-7335-4f53-8753-6eec8a51d775" (UID: "a0bd6584-7335-4f53-8753-6eec8a51d775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:54.082513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:54.082408    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:54.082513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:54.082454    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bd6584-7335-4f53-8753-6eec8a51d775-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:54.082513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:54.082468    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vgs8\" (UniqueName: \"kubernetes.io/projected/a0bd6584-7335-4f53-8753-6eec8a51d775-kube-api-access-8vgs8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:17:54.821086 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:54.821051    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"
Oct 30 04:17:54.821086 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:54.821080    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg" event={"ID":"a0bd6584-7335-4f53-8753-6eec8a51d775","Type":"ContainerDied","Data":"41d6294fb7e9d74a03075cb98f37b6abbf8b07277b8ac4745c676ba0376635cc"}
Oct 30 04:17:54.821356 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:54.821128    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41d6294fb7e9d74a03075cb98f37b6abbf8b07277b8ac4745c676ba0376635cc"
Oct 30 04:17:58.041968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:17:58.041936    4877 scope.go:122] "RemoveContainer" containerID="8482bb7d45b53d856d24a231e438b46f41d6f0a67a056421adce83322683b976"
Oct 30 04:18:02.026181 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:02.026141    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"]
Oct 30 04:18:02.035988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:02.035953    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xrcjq"]
Oct 30 04:18:03.697117 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:03.697080    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3708a65b-caf3-4e2e-bbec-f36f1505c2db" path="/var/lib/kubelet/pods/3708a65b-caf3-4e2e-bbec-f36f1505c2db/volumes"
Oct 30 04:18:06.867179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.867144    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"]
Oct 30 04:18:06.867649 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.867633    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a0bd6584-7335-4f53-8753-6eec8a51d775" containerName="revoke-keys"
Oct 30 04:18:06.867705 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.867650    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bd6584-7335-4f53-8753-6eec8a51d775" containerName="revoke-keys"
Oct 30 04:18:06.867743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.867724    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bd6584-7335-4f53-8753-6eec8a51d775" containerName="revoke-keys"
Oct 30 04:18:06.871130 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.871113    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:06.893897 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.893868    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 04:18:06.893897 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.893868    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vsjpp\""
Oct 30 04:18:06.897222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.897195    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"]
Oct 30 04:18:06.903601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.903575    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtccs\" (UniqueName: \"kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs\") pod \"payload-pre-processing-778cf68b57-lmcwx\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") " pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:06.903753 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:06.903612    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume\") pod \"payload-pre-processing-778cf68b57-lmcwx\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") " pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.005038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.004998    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtccs\" (UniqueName: \"kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs\") pod \"payload-pre-processing-778cf68b57-lmcwx\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") " pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.005038 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.005043    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume\") pod \"payload-pre-processing-778cf68b57-lmcwx\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") " pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.005855 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.005834    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume\") pod \"payload-pre-processing-778cf68b57-lmcwx\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") " pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.022013 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.021984    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtccs\" (UniqueName: \"kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs\") pod \"payload-pre-processing-778cf68b57-lmcwx\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") " pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.184339 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.184251    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.479811 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.479784    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"]
Oct 30 04:18:07.481649 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:18:07.481615    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1219d2_5431_4426_bf3f_d1bf24471d19.slice/crio-4c9d99875286a9dfde0ab84eaa3875d1358d477e3de6372cbb1a27693a4a5b3d WatchSource:0}: Error finding container 4c9d99875286a9dfde0ab84eaa3875d1358d477e3de6372cbb1a27693a4a5b3d: Status 404 returned error can't find the container with id 4c9d99875286a9dfde0ab84eaa3875d1358d477e3de6372cbb1a27693a4a5b3d
Oct 30 04:18:07.899380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.899338    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx" event={"ID":"7e1219d2-5431-4426-bf3f-d1bf24471d19","Type":"ContainerStarted","Data":"c4779fc4af9a3b79d816bb602076bec64b04a68f0860bd750e3f2731b5f698eb"}
Oct 30 04:18:07.899838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.899519    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:07.899838 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.899545    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx" event={"ID":"7e1219d2-5431-4426-bf3f-d1bf24471d19","Type":"ContainerStarted","Data":"4c9d99875286a9dfde0ab84eaa3875d1358d477e3de6372cbb1a27693a4a5b3d"}
Oct 30 04:18:07.935494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:07.935412    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx" podStartSLOduration=1.935392507 podStartE2EDuration="1.935392507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:07.932683352 +0000 UTC m=+4516.361572769" watchObservedRunningTime="2026-10-30 04:18:07.935392507 +0000 UTC m=+4516.364281923"
Oct 30 04:18:16.227854 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:18:16.227810    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:18:18.907939 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:18.907913    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:20.649192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.649154    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5797bb5d8c-cnmmz"]
Oct 30 04:18:20.653009 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.652987    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.672287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.672261    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5797bb5d8c-cnmmz"]
Oct 30 04:18:20.738987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.738953    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgmt\" (UniqueName: \"kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt\") pod \"payload-processing-5797bb5d8c-cnmmz\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") " pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.739140 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.739008    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume\") pod \"payload-processing-5797bb5d8c-cnmmz\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") " pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.840265 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.840212    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgmt\" (UniqueName: \"kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt\") pod \"payload-processing-5797bb5d8c-cnmmz\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") " pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.840457 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.840282    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume\") pod \"payload-processing-5797bb5d8c-cnmmz\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") " pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.841063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.841040    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume\") pod \"payload-processing-5797bb5d8c-cnmmz\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") " pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.858162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.858137    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgmt\" (UniqueName: \"kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt\") pod \"payload-processing-5797bb5d8c-cnmmz\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") " pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:20.965724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:20.965633    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:21.263381 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:21.263352    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5797bb5d8c-cnmmz"]
Oct 30 04:18:21.264543 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:18:21.264515    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973c3f36_7528_4ee4_b0cc_6b1aaf3f786c.slice/crio-e6205cff036513ef33875ca2bbfcc18dafea3c50d3fe3792284229233901d271 WatchSource:0}: Error finding container e6205cff036513ef33875ca2bbfcc18dafea3c50d3fe3792284229233901d271: Status 404 returned error can't find the container with id e6205cff036513ef33875ca2bbfcc18dafea3c50d3fe3792284229233901d271
Oct 30 04:18:21.991262 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:21.991222    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" event={"ID":"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c","Type":"ContainerStarted","Data":"cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52"}
Oct 30 04:18:21.991739 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:21.991319    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" event={"ID":"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c","Type":"ContainerStarted","Data":"e6205cff036513ef33875ca2bbfcc18dafea3c50d3fe3792284229233901d271"}
Oct 30 04:18:21.991739 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:21.991350    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:22.029174 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:22.029122    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" podStartSLOduration=2.029109943 podStartE2EDuration="2.029109943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:22.028308399 +0000 UTC m=+4530.457197817" watchObservedRunningTime="2026-10-30 04:18:22.029109943 +0000 UTC m=+4530.457999360"
Oct 30 04:18:27.025993 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:27.025963    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"]
Oct 30 04:18:27.035737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:27.035713    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qllm7"]
Oct 30 04:18:27.696637 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:27.696603    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c89ced-23ee-4e35-9dca-3ec6db57f477" path="/var/lib/kubelet/pods/57c89ced-23ee-4e35-9dca-3ec6db57f477/volumes"
Oct 30 04:18:33.001037 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:33.001003    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:43.034146 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:43.034061    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"]
Oct 30 04:18:43.043354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:43.043331    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6k66k"]
Oct 30 04:18:43.697010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:43.696974    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d4beab-4e48-41c3-9141-673143d8cf71" path="/var/lib/kubelet/pods/e7d4beab-4e48-41c3-9141-673143d8cf71/volumes"
Oct 30 04:18:46.228765 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:18:46.228723    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:18:47.656016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:47.655983    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5797bb5d8c-cnmmz"]
Oct 30 04:18:47.699608 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:47.699579    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"]
Oct 30 04:18:52.657002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:52.656929    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" podUID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" containerName="payload-processing" containerID="cri-o://cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52" gracePeriod=25
Oct 30 04:18:52.700538 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:52.700473    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx" podUID="7e1219d2-5431-4426-bf3f-d1bf24471d19" containerName="payload-pre-processing" containerID="cri-o://c4779fc4af9a3b79d816bb602076bec64b04a68f0860bd750e3f2731b5f698eb" gracePeriod=25
Oct 30 04:18:53.204990 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.204628    4877 generic.go:356] "Generic (PLEG): container finished" podID="7e1219d2-5431-4426-bf3f-d1bf24471d19" containerID="c4779fc4af9a3b79d816bb602076bec64b04a68f0860bd750e3f2731b5f698eb" exitCode=0
Oct 30 04:18:53.204990 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.204783    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx" event={"ID":"7e1219d2-5431-4426-bf3f-d1bf24471d19","Type":"ContainerDied","Data":"c4779fc4af9a3b79d816bb602076bec64b04a68f0860bd750e3f2731b5f698eb"}
Oct 30 04:18:53.211489 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.211354    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:53.224805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.217279    4877 generic.go:356] "Generic (PLEG): container finished" podID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" containerID="cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52" exitCode=0
Oct 30 04:18:53.224805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.217348    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" event={"ID":"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c","Type":"ContainerDied","Data":"cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52"}
Oct 30 04:18:53.224805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.217465    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" event={"ID":"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c","Type":"ContainerDied","Data":"e6205cff036513ef33875ca2bbfcc18dafea3c50d3fe3792284229233901d271"}
Oct 30 04:18:53.224805 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.217510    4877 scope.go:122] "RemoveContainer" containerID="cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52"
Oct 30 04:18:53.239251 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.239138    4877 scope.go:122] "RemoveContainer" containerID="cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52"
Oct 30 04:18:53.239598 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:18:53.239514    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52\": container with ID starting with cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52 not found: ID does not exist" containerID="cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52"
Oct 30 04:18:53.239598 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.239555    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52"} err="failed to get container status \"cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52\": rpc error: code = NotFound desc = could not find container \"cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52\": container with ID starting with cdf0c39c23065e08d5a86d4b4957944500efb848a7a6e224c769664ecbccbf52 not found: ID does not exist"
Oct 30 04:18:53.244675 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.244588    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.284187    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume\") pod \"7e1219d2-5431-4426-bf3f-d1bf24471d19\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") "
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.284237    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt\" (UniqueName: \"kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt\") pod \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") "
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.284299    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume\") pod \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\" (UID: \"973c3f36-7528-4ee4-b0cc-6b1aaf3f786c\") "
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.284405    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs\" (UniqueName: \"kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs\") pod \"7e1219d2-5431-4426-bf3f-d1bf24471d19\" (UID: \"7e1219d2-5431-4426-bf3f-d1bf24471d19\") "
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.289929    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume" pod "973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" (UID: "973c3f36-7528-4ee4-b0cc-6b1aaf3f786c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.290338    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume" pod "7e1219d2-5431-4426-bf3f-d1bf24471d19" (UID: "7e1219d2-5431-4426-bf3f-d1bf24471d19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:53.297644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.290683    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs" pod "7e1219d2-5431-4426-bf3f-d1bf24471d19" (UID: "7e1219d2-5431-4426-bf3f-d1bf24471d19"). InnerVolumeSpecName "kube-api-access-wtccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:53.304572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.304217    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt" pod "973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" (UID: "973c3f36-7528-4ee4-b0cc-6b1aaf3f786c"). InnerVolumeSpecName "kube-api-access-sqgmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:53.387452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.386042    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.387452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.386071    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtccs\" (UniqueName: \"kubernetes.io/projected/7e1219d2-5431-4426-bf3f-d1bf24471d19-kube-api-access-wtccs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.387452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.386084    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1219d2-5431-4426-bf3f-d1bf24471d19-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.387452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:53.386093    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgmt\" (UniqueName: \"kubernetes.io/projected/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c-kube-api-access-sqgmt\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:18:54.017888 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.017756    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz" podUID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.99:50052\" within 1s: context deadline exceeded"
Oct 30 04:18:54.017888 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:18:54.017861    4877 logging.go:55] [core] [Channel #16088 SubChannel #16089]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.99:50052", ServerName: "10.133.1.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.99:50052: operation was canceled"
Oct 30 04:18:54.027780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.027745    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"]
Oct 30 04:18:54.039004 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.037234    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-wjvfk"]
Oct 30 04:18:54.246448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.238207    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx" event={"ID":"7e1219d2-5431-4426-bf3f-d1bf24471d19","Type":"ContainerDied","Data":"4c9d99875286a9dfde0ab84eaa3875d1358d477e3de6372cbb1a27693a4a5b3d"}
Oct 30 04:18:54.246448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.238286    4877 scope.go:122] "RemoveContainer" containerID="c4779fc4af9a3b79d816bb602076bec64b04a68f0860bd750e3f2731b5f698eb"
Oct 30 04:18:54.246448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.238392    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"
Oct 30 04:18:54.252621 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.252600    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5797bb5d8c-cnmmz"
Oct 30 04:18:54.316028 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.315996    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5797bb5d8c-cnmmz"]
Oct 30 04:18:54.339841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.339810    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5797bb5d8c-cnmmz"]
Oct 30 04:18:54.375660 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.375627    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"]
Oct 30 04:18:54.380353 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:54.380289    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-778cf68b57-lmcwx"]
Oct 30 04:18:55.701222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:55.701183    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1219d2-5431-4426-bf3f-d1bf24471d19" path="/var/lib/kubelet/pods/7e1219d2-5431-4426-bf3f-d1bf24471d19/volumes"
Oct 30 04:18:55.701824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:55.701777    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" path="/var/lib/kubelet/pods/973c3f36-7528-4ee4-b0cc-6b1aaf3f786c/volumes"
Oct 30 04:18:55.702332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:55.702305    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdd374c-d218-484e-ad5d-2724aefca163" path="/var/lib/kubelet/pods/9bdd374c-d218-484e-ad5d-2724aefca163/volumes"
Oct 30 04:18:58.068471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:18:58.068416    4877 scope.go:122] "RemoveContainer" containerID="7e79a9ca7c0d4e587a0315996329f0d832791291916e51be9d1fefb73c3926fa"
Oct 30 04:19:16.228760 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:19:16.228720    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:19:46.228057 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:19:46.228017    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:19:58.100307 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:19:58.100262    4877 scope.go:122] "RemoveContainer" containerID="62fdb1a3a009b96aab21e54cf96bd5191d92895d39928e4d39f3776d5a4b0a5b"
Oct 30 04:19:58.109535 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:19:58.109513    4877 scope.go:122] "RemoveContainer" containerID="15a2bb2430f0f5e42a4e783545d41d87a74e9f4c08c2f69af62adc003f60d91b"
Oct 30 04:19:58.117545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:19:58.117526    4877 scope.go:122] "RemoveContainer" containerID="2efcf9b9a3aa2afbcb1b526c6445fb0dd069d84a9dc6fb046cd582a2eb0607b1"
Oct 30 04:19:58.125680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:19:58.125662    4877 scope.go:122] "RemoveContainer" containerID="348bdc1311f8a41c78c76d86c41913e4351924fa4b565f66d95d2de28c1545fa"
Oct 30 04:20:13.031916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:13.031826    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"]
Oct 30 04:20:13.041683 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:13.041645    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68394a88-v22hz"]
Oct 30 04:20:13.696614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:13.696580    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75710f4a-5000-4755-9a24-381fbdfd0188" path="/var/lib/kubelet/pods/75710f4a-5000-4755-9a24-381fbdfd0188/volumes"
Oct 30 04:20:16.229052 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:20:16.229013    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:20:29.470011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.469975    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"]
Oct 30 04:20:29.470665 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.470639    4877 cpu_manager.go:414] "containerMap: removing container" podUID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" containerName="payload-processing"
Oct 30 04:20:29.470665 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.470659    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" containerName="payload-processing"
Oct 30 04:20:29.470809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.470703    4877 cpu_manager.go:414] "containerMap: removing container" podUID="7e1219d2-5431-4426-bf3f-d1bf24471d19" containerName="payload-pre-processing"
Oct 30 04:20:29.470809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.470712    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1219d2-5431-4426-bf3f-d1bf24471d19" containerName="payload-pre-processing"
Oct 30 04:20:29.470916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.470854    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1219d2-5431-4426-bf3f-d1bf24471d19" containerName="payload-pre-processing"
Oct 30 04:20:29.470916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.470875    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="973c3f36-7528-4ee4-b0cc-6b1aaf3f786c" containerName="payload-processing"
Oct 30 04:20:29.473801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.473780    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.496153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.496122    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:20:29.496153 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.496136    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:20:29.499408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.499385    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"]
Oct 30 04:20:29.656828 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.656792    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.657022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.656838    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhwr\" (UniqueName: \"kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.657022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.656973    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.758537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.758405    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.758537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.758490    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhwr\" (UniqueName: \"kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.758786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.758557    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.759095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.759069    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.760960 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.760939    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.776593 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.776565    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhwr\" (UniqueName: \"kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr\") pod \"maas-api-revoke-sub-e2547622-mxj2w\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:29.786933 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:29.786910    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:30.086728 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:30.086691    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"]
Oct 30 04:20:30.089342 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:20:30.089307    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d2fc9c_865b_47c9_996b_1c627debbf8f.slice/crio-95a95377c37ed31b58026f9c03b1581e34cf60cf2137f507c5a70fcc9eee68c0 WatchSource:0}: Error finding container 95a95377c37ed31b58026f9c03b1581e34cf60cf2137f507c5a70fcc9eee68c0: Status 404 returned error can't find the container with id 95a95377c37ed31b58026f9c03b1581e34cf60cf2137f507c5a70fcc9eee68c0
Oct 30 04:20:30.875100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:30.875061    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w" event={"ID":"d0d2fc9c-865b-47c9-996b-1c627debbf8f","Type":"ContainerStarted","Data":"ed8636842886d9d1643f559195585e80f558887528e78e7aefe3060c675a14ef"}
Oct 30 04:20:30.875538 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:30.875128    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w" event={"ID":"d0d2fc9c-865b-47c9-996b-1c627debbf8f","Type":"ContainerStarted","Data":"95a95377c37ed31b58026f9c03b1581e34cf60cf2137f507c5a70fcc9eee68c0"}
Oct 30 04:20:30.911058 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:30.911017    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w" podStartSLOduration=1.911004198 podStartE2EDuration="1.911004198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:30.910231364 +0000 UTC m=+4659.339120802" watchObservedRunningTime="2026-10-30 04:20:30.911004198 +0000 UTC m=+4659.339893609"
Oct 30 04:20:31.881992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:31.881954    4877 generic.go:356] "Generic (PLEG): container finished" podID="d0d2fc9c-865b-47c9-996b-1c627debbf8f" containerID="ed8636842886d9d1643f559195585e80f558887528e78e7aefe3060c675a14ef" exitCode=0
Oct 30 04:20:31.882393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:31.882047    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w" event={"ID":"d0d2fc9c-865b-47c9-996b-1c627debbf8f","Type":"ContainerDied","Data":"ed8636842886d9d1643f559195585e80f558887528e78e7aefe3060c675a14ef"}
Oct 30 04:20:33.033308 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.033284    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:33.088486 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.088457    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca\") pod \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") "
Oct 30 04:20:33.088637 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.088563    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr\" (UniqueName: \"kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr\") pod \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") "
Oct 30 04:20:33.088702 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.088683    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca\") pod \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\" (UID: \"d0d2fc9c-865b-47c9-996b-1c627debbf8f\") "
Oct 30 04:20:33.089141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.089108    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca" pod "d0d2fc9c-865b-47c9-996b-1c627debbf8f" (UID: "d0d2fc9c-865b-47c9-996b-1c627debbf8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:33.090587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.090565    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca" pod "d0d2fc9c-865b-47c9-996b-1c627debbf8f" (UID: "d0d2fc9c-865b-47c9-996b-1c627debbf8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:33.090670 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.090656    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr" pod "d0d2fc9c-865b-47c9-996b-1c627debbf8f" (UID: "d0d2fc9c-865b-47c9-996b-1c627debbf8f"). InnerVolumeSpecName "kube-api-access-wwhwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:33.190268 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.190194    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:20:33.190268 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.190222    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhwr\" (UniqueName: \"kubernetes.io/projected/d0d2fc9c-865b-47c9-996b-1c627debbf8f-kube-api-access-wwhwr\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:20:33.190268 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.190236    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d2fc9c-865b-47c9-996b-1c627debbf8f-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:20:33.893899 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.893858    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w" event={"ID":"d0d2fc9c-865b-47c9-996b-1c627debbf8f","Type":"ContainerDied","Data":"95a95377c37ed31b58026f9c03b1581e34cf60cf2137f507c5a70fcc9eee68c0"}
Oct 30 04:20:33.894083 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.893914    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a95377c37ed31b58026f9c03b1581e34cf60cf2137f507c5a70fcc9eee68c0"
Oct 30 04:20:33.894083 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:33.893878    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"
Oct 30 04:20:46.228151 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:20:46.228109    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:20:50.882555 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:50.882512    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:20:50.883202 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:50.883164    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:20:51.313448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:51.312602    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:20:51.313448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:51.312679    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:20:52.724955 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.724890    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"]
Oct 30 04:20:52.725706 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.725684    4877 cpu_manager.go:414] "containerMap: removing container" podUID="d0d2fc9c-865b-47c9-996b-1c627debbf8f" containerName="revoke-keys"
Oct 30 04:20:52.725771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.725710    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d2fc9c-865b-47c9-996b-1c627debbf8f" containerName="revoke-keys"
Oct 30 04:20:52.725865 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.725851    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d2fc9c-865b-47c9-996b-1c627debbf8f" containerName="revoke-keys"
Oct 30 04:20:52.729703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.729682    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.747352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.747323    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"]
Oct 30 04:20:52.754008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.753981    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:20:52.754149 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.754009    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:20:52.861221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.861181    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbzxj\" (UniqueName: \"kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.861398 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.861360    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.861482 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.861397    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.962165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.962126    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.962361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.962173    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.962361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.962225    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbzxj\" (UniqueName: \"kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.962929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.962910    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.964548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.964526    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:52.979357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:52.979295    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbzxj\" (UniqueName: \"kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj\") pod \"maas-api-revoke-sub-c1e271a9-xbjnt\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:53.042950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:53.042924    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:53.340886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:53.340858    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"]
Oct 30 04:20:53.342656 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:20:53.342627    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1712fa6a_3a6c_4bea_8db2_6b0278499014.slice/crio-7f6dd4da12000ccfcdf175fe66a1a9d8cfdf53c402250db3da9d34786aa5d206 WatchSource:0}: Error finding container 7f6dd4da12000ccfcdf175fe66a1a9d8cfdf53c402250db3da9d34786aa5d206: Status 404 returned error can't find the container with id 7f6dd4da12000ccfcdf175fe66a1a9d8cfdf53c402250db3da9d34786aa5d206
Oct 30 04:20:54.075508 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:54.075468    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt" event={"ID":"1712fa6a-3a6c-4bea-8db2-6b0278499014","Type":"ContainerStarted","Data":"6a1195e2dba9bf3668f4faf81d89e0fc355a50a7f23815d857dafafe37439440"}
Oct 30 04:20:54.075966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:54.075529    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt" event={"ID":"1712fa6a-3a6c-4bea-8db2-6b0278499014","Type":"ContainerStarted","Data":"7f6dd4da12000ccfcdf175fe66a1a9d8cfdf53c402250db3da9d34786aa5d206"}
Oct 30 04:20:54.118069 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:54.118024    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt" podStartSLOduration=2.11800939 podStartE2EDuration="2.11800939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:54.115819428 +0000 UTC m=+4682.544708846" watchObservedRunningTime="2026-10-30 04:20:54.11800939 +0000 UTC m=+4682.546898804"
Oct 30 04:20:55.083403 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:55.083363    4877 generic.go:356] "Generic (PLEG): container finished" podID="1712fa6a-3a6c-4bea-8db2-6b0278499014" containerID="6a1195e2dba9bf3668f4faf81d89e0fc355a50a7f23815d857dafafe37439440" exitCode=0
Oct 30 04:20:55.083827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:55.083453    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt" event={"ID":"1712fa6a-3a6c-4bea-8db2-6b0278499014","Type":"ContainerDied","Data":"6a1195e2dba9bf3668f4faf81d89e0fc355a50a7f23815d857dafafe37439440"}
Oct 30 04:20:56.218726 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.218702    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:56.393979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.393890    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca\") pod \"1712fa6a-3a6c-4bea-8db2-6b0278499014\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") "
Oct 30 04:20:56.393979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.393940    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj\" (UniqueName: \"kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj\") pod \"1712fa6a-3a6c-4bea-8db2-6b0278499014\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") "
Oct 30 04:20:56.393979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.393972    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca\") pod \"1712fa6a-3a6c-4bea-8db2-6b0278499014\" (UID: \"1712fa6a-3a6c-4bea-8db2-6b0278499014\") "
Oct 30 04:20:56.394361 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.394339    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca" pod "1712fa6a-3a6c-4bea-8db2-6b0278499014" (UID: "1712fa6a-3a6c-4bea-8db2-6b0278499014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:56.395818 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.395787    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca" pod "1712fa6a-3a6c-4bea-8db2-6b0278499014" (UID: "1712fa6a-3a6c-4bea-8db2-6b0278499014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:56.395936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.395878    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj" pod "1712fa6a-3a6c-4bea-8db2-6b0278499014" (UID: "1712fa6a-3a6c-4bea-8db2-6b0278499014"). InnerVolumeSpecName "kube-api-access-jbzxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:56.495677 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.495645    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:20:56.495677 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.495673    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbzxj\" (UniqueName: \"kubernetes.io/projected/1712fa6a-3a6c-4bea-8db2-6b0278499014-kube-api-access-jbzxj\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:20:56.495677 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:56.495683    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1712fa6a-3a6c-4bea-8db2-6b0278499014-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:20:57.095898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:57.095863    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"
Oct 30 04:20:57.095898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:57.095879    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt" event={"ID":"1712fa6a-3a6c-4bea-8db2-6b0278499014","Type":"ContainerDied","Data":"7f6dd4da12000ccfcdf175fe66a1a9d8cfdf53c402250db3da9d34786aa5d206"}
Oct 30 04:20:57.096129 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:57.095923    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6dd4da12000ccfcdf175fe66a1a9d8cfdf53c402250db3da9d34786aa5d206"
Oct 30 04:20:58.160841 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:20:58.160807    4877 scope.go:122] "RemoveContainer" containerID="58952b04a4270753fce85e6d86204f68731fd69dfa7959b3c509725a69e1d296"
Oct 30 04:21:11.791614 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:11.791565    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fl76j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:21:11.792147 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:11.791644    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fl76j" podUID="d7badd71-7851-4ddb-b54b-86d2707078be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:21:12.666139 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.666095    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"]
Oct 30 04:21:12.666668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.666644    4877 cpu_manager.go:414] "containerMap: removing container" podUID="1712fa6a-3a6c-4bea-8db2-6b0278499014" containerName="revoke-keys"
Oct 30 04:21:12.666784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.666669    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1712fa6a-3a6c-4bea-8db2-6b0278499014" containerName="revoke-keys"
Oct 30 04:21:12.666847 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.666815    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="1712fa6a-3a6c-4bea-8db2-6b0278499014" containerName="revoke-keys"
Oct 30 04:21:12.670844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.670814    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.691141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.691112    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"]
Oct 30 04:21:12.696025 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.695999    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:21:12.696158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.696081    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:21:12.779821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.779779    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.780030 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.779834    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8r6\" (UniqueName: \"kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.780030 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.779972    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.881126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.881080    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.881668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.881191    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.881668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.881237    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8r6\" (UniqueName: \"kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.881800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.881780    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.883668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.883636    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.897971 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.897949    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8r6\" (UniqueName: \"kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6\") pod \"maas-api-revoke-sub-2e1f5b5f-fzpm9\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:12.984695 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:12.984616    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:13.283633 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:13.283603    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"]
Oct 30 04:21:13.285492 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:21:13.285457    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40012a48_1108_4b10_a9e2_f58e225bad91.slice/crio-3511d45c6107b934416ae02752e9a3886700ffb2ce9856cebc5369e3b63af776 WatchSource:0}: Error finding container 3511d45c6107b934416ae02752e9a3886700ffb2ce9856cebc5369e3b63af776: Status 404 returned error can't find the container with id 3511d45c6107b934416ae02752e9a3886700ffb2ce9856cebc5369e3b63af776
Oct 30 04:21:14.225485 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:14.225448    4877 generic.go:356] "Generic (PLEG): container finished" podID="40012a48-1108-4b10-a9e2-f58e225bad91" containerID="cefd832c3e042ee37b82f7f20919bde1ea4cd61b2110da4289b1ef1c1af3977a" exitCode=0
Oct 30 04:21:14.225885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:14.225525    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9" event={"ID":"40012a48-1108-4b10-a9e2-f58e225bad91","Type":"ContainerDied","Data":"cefd832c3e042ee37b82f7f20919bde1ea4cd61b2110da4289b1ef1c1af3977a"}
Oct 30 04:21:14.225885 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:14.225579    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9" event={"ID":"40012a48-1108-4b10-a9e2-f58e225bad91","Type":"ContainerStarted","Data":"3511d45c6107b934416ae02752e9a3886700ffb2ce9856cebc5369e3b63af776"}
Oct 30 04:21:15.377548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.377525    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:15.404822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.404791    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6\" (UniqueName: \"kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6\") pod \"40012a48-1108-4b10-a9e2-f58e225bad91\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") "
Oct 30 04:21:15.404965 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.404877    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca\") pod \"40012a48-1108-4b10-a9e2-f58e225bad91\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") "
Oct 30 04:21:15.405022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.404972    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca\") pod \"40012a48-1108-4b10-a9e2-f58e225bad91\" (UID: \"40012a48-1108-4b10-a9e2-f58e225bad91\") "
Oct 30 04:21:15.405350 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.405322    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca" pod "40012a48-1108-4b10-a9e2-f58e225bad91" (UID: "40012a48-1108-4b10-a9e2-f58e225bad91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:15.406867 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.406835    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca" pod "40012a48-1108-4b10-a9e2-f58e225bad91" (UID: "40012a48-1108-4b10-a9e2-f58e225bad91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:15.406968 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.406902    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6" pod "40012a48-1108-4b10-a9e2-f58e225bad91" (UID: "40012a48-1108-4b10-a9e2-f58e225bad91"). InnerVolumeSpecName "kube-api-access-4p8r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:15.506185 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.506100    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:15.506185 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.506131    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40012a48-1108-4b10-a9e2-f58e225bad91-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:15.506185 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:15.506141    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8r6\" (UniqueName: \"kubernetes.io/projected/40012a48-1108-4b10-a9e2-f58e225bad91-kube-api-access-4p8r6\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:16.227989 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:21:16.227950    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:21:16.242790 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:16.242743    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9" event={"ID":"40012a48-1108-4b10-a9e2-f58e225bad91","Type":"ContainerDied","Data":"3511d45c6107b934416ae02752e9a3886700ffb2ce9856cebc5369e3b63af776"}
Oct 30 04:21:16.242988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:16.242804    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"
Oct 30 04:21:16.242988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:16.242857    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3511d45c6107b934416ae02752e9a3886700ffb2ce9856cebc5369e3b63af776"
Oct 30 04:21:17.024927 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:17.024896    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"]
Oct 30 04:21:17.034632 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:17.034607    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e251bc8d-wpl2h"]
Oct 30 04:21:17.696590 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:17.696559    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f3d084-ef78-4333-84eb-01e36687b2cb" path="/var/lib/kubelet/pods/c1f3d084-ef78-4333-84eb-01e36687b2cb/volumes"
Oct 30 04:21:23.883752 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.883709    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"]
Oct 30 04:21:23.884188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.884174    4877 cpu_manager.go:414] "containerMap: removing container" podUID="40012a48-1108-4b10-a9e2-f58e225bad91" containerName="revoke-keys"
Oct 30 04:21:23.884236 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.884190    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40012a48-1108-4b10-a9e2-f58e225bad91" containerName="revoke-keys"
Oct 30 04:21:23.884297 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.884287    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="40012a48-1108-4b10-a9e2-f58e225bad91" containerName="revoke-keys"
Oct 30 04:21:23.887861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.887842    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:23.905617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.905585    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"]
Oct 30 04:21:23.912631 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.912606    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:21:23.912824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.912808    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:21:23.989317 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.989285    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:23.989505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.989321    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:23.989505 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:23.989368    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsm8\" (UniqueName: \"kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.090330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.090289    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.090330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.090330    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.090638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.090370    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsm8\" (UniqueName: \"kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.091008 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.090983    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.092878 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.092846    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.107487 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.107465    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsm8\" (UniqueName: \"kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8\") pod \"maas-api-revoke-sub-ec843018-vfbhw\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.201037 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.200945    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:24.503950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:24.503924    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"]
Oct 30 04:21:24.506365 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:21:24.506336    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ebf4d8_ade5_4033_8d21_b6ea8101e3eb.slice/crio-38cb250ecafa6efed9bf9a08543c2f29ec37bdc3013d5bf73e250239dccb1f05 WatchSource:0}: Error finding container 38cb250ecafa6efed9bf9a08543c2f29ec37bdc3013d5bf73e250239dccb1f05: Status 404 returned error can't find the container with id 38cb250ecafa6efed9bf9a08543c2f29ec37bdc3013d5bf73e250239dccb1f05
Oct 30 04:21:25.296966 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:25.296923    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw" event={"ID":"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb","Type":"ContainerStarted","Data":"07084438a4ed706231b6ef828fe13c4e9fc9d51a641510d9ec0dd94c9ba381bf"}
Oct 30 04:21:25.297503 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:25.296990    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw" event={"ID":"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb","Type":"ContainerStarted","Data":"38cb250ecafa6efed9bf9a08543c2f29ec37bdc3013d5bf73e250239dccb1f05"}
Oct 30 04:21:25.331668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:25.331615    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw" podStartSLOduration=2.331601544 podStartE2EDuration="2.331601544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:25.33023955 +0000 UTC m=+4713.759128989" watchObservedRunningTime="2026-10-30 04:21:25.331601544 +0000 UTC m=+4713.760490962"
Oct 30 04:21:26.304991 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:26.304944    4877 generic.go:356] "Generic (PLEG): container finished" podID="29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" containerID="07084438a4ed706231b6ef828fe13c4e9fc9d51a641510d9ec0dd94c9ba381bf" exitCode=0
Oct 30 04:21:26.305385 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:26.305037    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw" event={"ID":"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb","Type":"ContainerDied","Data":"07084438a4ed706231b6ef828fe13c4e9fc9d51a641510d9ec0dd94c9ba381bf"}
Oct 30 04:21:27.462372 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.462338    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:27.526089 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.526060    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca\") pod \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") "
Oct 30 04:21:27.526260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.526212    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8\" (UniqueName: \"kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8\") pod \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") "
Oct 30 04:21:27.526260 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.526252    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca\") pod \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\" (UID: \"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb\") "
Oct 30 04:21:27.526530 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.526492    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca" pod "29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" (UID: "29ebf4d8-ade5-4033-8d21-b6ea8101e3eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:27.528363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.528331    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8" pod "29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" (UID: "29ebf4d8-ade5-4033-8d21-b6ea8101e3eb"). InnerVolumeSpecName "kube-api-access-kqsm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:27.528540 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.528520    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca" pod "29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" (UID: "29ebf4d8-ade5-4033-8d21-b6ea8101e3eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:27.627170 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.627087    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqsm8\" (UniqueName: \"kubernetes.io/projected/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-kube-api-access-kqsm8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:27.627170 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.627119    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:27.627170 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:27.627128    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:28.319572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:28.319536    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"
Oct 30 04:21:28.319572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:28.319555    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw" event={"ID":"29ebf4d8-ade5-4033-8d21-b6ea8101e3eb","Type":"ContainerDied","Data":"38cb250ecafa6efed9bf9a08543c2f29ec37bdc3013d5bf73e250239dccb1f05"}
Oct 30 04:21:28.319794 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:28.319604    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38cb250ecafa6efed9bf9a08543c2f29ec37bdc3013d5bf73e250239dccb1f05"
Oct 30 04:21:46.228759 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:21:46.228670    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:21:52.055771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.055734    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"]
Oct 30 04:21:52.056210 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.056196    4877 cpu_manager.go:414] "containerMap: removing container" podUID="29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" containerName="revoke-keys"
Oct 30 04:21:52.056256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.056212    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" containerName="revoke-keys"
Oct 30 04:21:52.056319 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.056309    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" containerName="revoke-keys"
Oct 30 04:21:52.061287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.061267    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.076375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.076349    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:21:52.079419 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.079393    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"]
Oct 30 04:21:52.086511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.086484    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:21:52.181521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.181486    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.181737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.181551    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwb7\" (UniqueName: \"kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.181737 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.181633    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.283032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.282978    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwb7\" (UniqueName: \"kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.283032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.283036    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.283283 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.283109    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.283789 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.283768    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.285581 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.285562    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.300332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.300304    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwb7\" (UniqueName: \"kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7\") pod \"maas-api-revoke-sub-046e1343-t4qj6\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.374713 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.374626    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:52.673882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:52.673579    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"]
Oct 30 04:21:52.675751 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:21:52.675711    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ec346d_05bb_4c33_a7ad_0057297e8727.slice/crio-833971e53ab275d85091ddd3a35ef3ac1ce8f0c321ac1c159fc9be495adab566 WatchSource:0}: Error finding container 833971e53ab275d85091ddd3a35ef3ac1ce8f0c321ac1c159fc9be495adab566: Status 404 returned error can't find the container with id 833971e53ab275d85091ddd3a35ef3ac1ce8f0c321ac1c159fc9be495adab566
Oct 30 04:21:53.521619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:53.521577    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6" event={"ID":"a5ec346d-05bb-4c33-a7ad-0057297e8727","Type":"ContainerStarted","Data":"f80174e65838dfd01eedbb9545351f213ace938da732237edeb4ab6fda63920c"}
Oct 30 04:21:53.522023 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:53.521634    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6" event={"ID":"a5ec346d-05bb-4c33-a7ad-0057297e8727","Type":"ContainerStarted","Data":"833971e53ab275d85091ddd3a35ef3ac1ce8f0c321ac1c159fc9be495adab566"}
Oct 30 04:21:53.556270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:53.556210    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6" podStartSLOduration=1.5561939740000001 podStartE2EDuration="1.556193974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:53.55490443 +0000 UTC m=+4741.983793848" watchObservedRunningTime="2026-10-30 04:21:53.556193974 +0000 UTC m=+4741.985083392"
Oct 30 04:21:54.529337 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:54.529302    4877 generic.go:356] "Generic (PLEG): container finished" podID="a5ec346d-05bb-4c33-a7ad-0057297e8727" containerID="f80174e65838dfd01eedbb9545351f213ace938da732237edeb4ab6fda63920c" exitCode=0
Oct 30 04:21:54.529775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:54.529371    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6" event={"ID":"a5ec346d-05bb-4c33-a7ad-0057297e8727","Type":"ContainerDied","Data":"f80174e65838dfd01eedbb9545351f213ace938da732237edeb4ab6fda63920c"}
Oct 30 04:21:55.673357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.673331    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:55.819884 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.819796    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7\" (UniqueName: \"kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7\") pod \"a5ec346d-05bb-4c33-a7ad-0057297e8727\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") "
Oct 30 04:21:55.820068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.819898    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca\") pod \"a5ec346d-05bb-4c33-a7ad-0057297e8727\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") "
Oct 30 04:21:55.820068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.819949    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca\") pod \"a5ec346d-05bb-4c33-a7ad-0057297e8727\" (UID: \"a5ec346d-05bb-4c33-a7ad-0057297e8727\") "
Oct 30 04:21:55.820323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.820294    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca" pod "a5ec346d-05bb-4c33-a7ad-0057297e8727" (UID: "a5ec346d-05bb-4c33-a7ad-0057297e8727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:55.821961 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.821938    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca" pod "a5ec346d-05bb-4c33-a7ad-0057297e8727" (UID: "a5ec346d-05bb-4c33-a7ad-0057297e8727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:55.821961 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.821948    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7" pod "a5ec346d-05bb-4c33-a7ad-0057297e8727" (UID: "a5ec346d-05bb-4c33-a7ad-0057297e8727"). InnerVolumeSpecName "kube-api-access-jmwb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:55.921243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.921199    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:55.921243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.921241    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwb7\" (UniqueName: \"kubernetes.io/projected/a5ec346d-05bb-4c33-a7ad-0057297e8727-kube-api-access-jmwb7\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:55.921243 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:55.921253    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ec346d-05bb-4c33-a7ad-0057297e8727-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:21:56.543151 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:56.543120    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"
Oct 30 04:21:56.543352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:56.543151    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6" event={"ID":"a5ec346d-05bb-4c33-a7ad-0057297e8727","Type":"ContainerDied","Data":"833971e53ab275d85091ddd3a35ef3ac1ce8f0c321ac1c159fc9be495adab566"}
Oct 30 04:21:56.543352 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:56.543204    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833971e53ab275d85091ddd3a35ef3ac1ce8f0c321ac1c159fc9be495adab566"
Oct 30 04:21:58.186645 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:21:58.186609    4877 scope.go:122] "RemoveContainer" containerID="cab033d98319b82872559fab8b24d2794e66aa327935fb932d13026ff8de8110"
Oct 30 04:22:02.420351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.420309    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"]
Oct 30 04:22:02.420893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.420755    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a5ec346d-05bb-4c33-a7ad-0057297e8727" containerName="revoke-keys"
Oct 30 04:22:02.420893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.420768    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ec346d-05bb-4c33-a7ad-0057297e8727" containerName="revoke-keys"
Oct 30 04:22:02.420893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.420864    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ec346d-05bb-4c33-a7ad-0057297e8727" containerName="revoke-keys"
Oct 30 04:22:02.425440 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.425404    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.450455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.450381    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:22:02.450455 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.450381    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:22:02.455445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.455397    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"]
Oct 30 04:22:02.476772 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.476733    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgwv\" (UniqueName: \"kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.476972 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.476879    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.476972 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.476949    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.578473 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.578405    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.578682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.578516    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.578682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.578585    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgwv\" (UniqueName: \"kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.579169 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.579143    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.581015 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.580986    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.595899 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.595869    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgwv\" (UniqueName: \"kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv\") pod \"maas-api-revoke-sub-d3fba056-8t4s7\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:02.740138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:02.740039    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:03.047398 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:03.047367    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"]
Oct 30 04:22:03.049319 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:22:03.049289    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f7ee7f_688c_4560_8621_be77e605fe4d.slice/crio-ee1e028cb2f925e47dc37e2c5825086d83dfdcab19cf0c9ca2c161d26c24e2ce WatchSource:0}: Error finding container ee1e028cb2f925e47dc37e2c5825086d83dfdcab19cf0c9ca2c161d26c24e2ce: Status 404 returned error can't find the container with id ee1e028cb2f925e47dc37e2c5825086d83dfdcab19cf0c9ca2c161d26c24e2ce
Oct 30 04:22:03.591230 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:03.591192    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7" event={"ID":"69f7ee7f-688c-4560-8621-be77e605fe4d","Type":"ContainerStarted","Data":"154dcc2781fa40ab442662f9374eced0cc3bdd96dde8774017cbedf09d94fa0b"}
Oct 30 04:22:03.591631 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:03.591261    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7" event={"ID":"69f7ee7f-688c-4560-8621-be77e605fe4d","Type":"ContainerStarted","Data":"ee1e028cb2f925e47dc37e2c5825086d83dfdcab19cf0c9ca2c161d26c24e2ce"}
Oct 30 04:22:03.626997 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:03.626945    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7" podStartSLOduration=1.6269314339999998 podStartE2EDuration="1.626931434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:03.625727768 +0000 UTC m=+4752.054617186" watchObservedRunningTime="2026-10-30 04:22:03.626931434 +0000 UTC m=+4752.055820852"
Oct 30 04:22:04.603986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:04.603945    4877 generic.go:356] "Generic (PLEG): container finished" podID="69f7ee7f-688c-4560-8621-be77e605fe4d" containerID="154dcc2781fa40ab442662f9374eced0cc3bdd96dde8774017cbedf09d94fa0b" exitCode=0
Oct 30 04:22:04.604535 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:04.604041    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7" event={"ID":"69f7ee7f-688c-4560-8621-be77e605fe4d","Type":"ContainerDied","Data":"154dcc2781fa40ab442662f9374eced0cc3bdd96dde8774017cbedf09d94fa0b"}
Oct 30 04:22:05.904527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:05.886533    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:22:05.904527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:05.886602    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:22:06.150848 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.150812    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:06.176517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.176403    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca\") pod \"69f7ee7f-688c-4560-8621-be77e605fe4d\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") "
Oct 30 04:22:06.176809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.176785    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv\" (UniqueName: \"kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv\") pod \"69f7ee7f-688c-4560-8621-be77e605fe4d\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") "
Oct 30 04:22:06.176993 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.176974    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca\") pod \"69f7ee7f-688c-4560-8621-be77e605fe4d\" (UID: \"69f7ee7f-688c-4560-8621-be77e605fe4d\") "
Oct 30 04:22:06.196621 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.193590    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca" pod "69f7ee7f-688c-4560-8621-be77e605fe4d" (UID: "69f7ee7f-688c-4560-8621-be77e605fe4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:22:06.203597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.203554    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca" pod "69f7ee7f-688c-4560-8621-be77e605fe4d" (UID: "69f7ee7f-688c-4560-8621-be77e605fe4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:22:06.204188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.204165    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv" pod "69f7ee7f-688c-4560-8621-be77e605fe4d" (UID: "69f7ee7f-688c-4560-8621-be77e605fe4d"). InnerVolumeSpecName "kube-api-access-9vgwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:22:06.285607 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.285559    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:06.285814 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.285636    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f7ee7f-688c-4560-8621-be77e605fe4d-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:06.285814 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.285660    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgwv\" (UniqueName: \"kubernetes.io/projected/69f7ee7f-688c-4560-8621-be77e605fe4d-kube-api-access-9vgwv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:06.316996 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.316516    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:22:06.316996 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.316582    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:22:06.631189 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.631151    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"
Oct 30 04:22:06.631189 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.631172    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7" event={"ID":"69f7ee7f-688c-4560-8621-be77e605fe4d","Type":"ContainerDied","Data":"ee1e028cb2f925e47dc37e2c5825086d83dfdcab19cf0c9ca2c161d26c24e2ce"}
Oct 30 04:22:06.631418 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:06.631228    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1e028cb2f925e47dc37e2c5825086d83dfdcab19cf0c9ca2c161d26c24e2ce"
Oct 30 04:22:16.229636 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:22:16.229587    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:22:19.030998 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:19.030945    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"]
Oct 30 04:22:19.040343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:19.040313    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dlhgv"]
Oct 30 04:22:19.699810 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:19.699766    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e496f11-ad9b-450e-ae36-e5ea8f1c32e1" path="/var/lib/kubelet/pods/0e496f11-ad9b-450e-ae36-e5ea8f1c32e1/volumes"
Oct 30 04:22:20.887603 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:20.887554    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:22:20.888273 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:20.888219    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:22:21.303448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:21.303036    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:22:21.303448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:21.303137    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:22:24.263245 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.263208    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"]
Oct 30 04:22:24.263685 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.263630    4877 cpu_manager.go:414] "containerMap: removing container" podUID="69f7ee7f-688c-4560-8621-be77e605fe4d" containerName="revoke-keys"
Oct 30 04:22:24.263685 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.263641    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f7ee7f-688c-4560-8621-be77e605fe4d" containerName="revoke-keys"
Oct 30 04:22:24.263766 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.263732    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f7ee7f-688c-4560-8621-be77e605fe4d" containerName="revoke-keys"
Oct 30 04:22:24.267168 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.267153    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.279389 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.279366    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:22:24.289332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.289308    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:22:24.292852 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.292823    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"]
Oct 30 04:22:24.388494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.388455    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn28l\" (UniqueName: \"kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.388494 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.388497    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.388752 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.388571    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.489905 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.489868    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn28l\" (UniqueName: \"kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.490078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.489913    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.490078 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.490026    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.490716 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.490688    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.492224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.492201    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.506728 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.506704    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn28l\" (UniqueName: \"kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l\") pod \"maas-api-revoke-sub-ae594979-55gpv\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.580583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.580549    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:24.875196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:24.875160    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"]
Oct 30 04:22:24.876881 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:22:24.876852    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cbbb03_08c9_4fb7_b117_e0cc92eb3f5c.slice/crio-1c5bda3c6b647a11e81b3a23b651f406dbcb3cb202e225e8e3d32858d8842745 WatchSource:0}: Error finding container 1c5bda3c6b647a11e81b3a23b651f406dbcb3cb202e225e8e3d32858d8842745: Status 404 returned error can't find the container with id 1c5bda3c6b647a11e81b3a23b651f406dbcb3cb202e225e8e3d32858d8842745
Oct 30 04:22:25.760946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:25.760908    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv" event={"ID":"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c","Type":"ContainerStarted","Data":"b210a18b4085f91c99c4a00560217f571c275805ed949b4ef77c6f69ee5e2853"}
Oct 30 04:22:25.761370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:25.760971    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv" event={"ID":"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c","Type":"ContainerStarted","Data":"1c5bda3c6b647a11e81b3a23b651f406dbcb3cb202e225e8e3d32858d8842745"}
Oct 30 04:22:25.796396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:25.796330    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv" podStartSLOduration=1.796316976 podStartE2EDuration="1.796316976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:25.794823108 +0000 UTC m=+4774.223712526" watchObservedRunningTime="2026-10-30 04:22:25.796316976 +0000 UTC m=+4774.225206385"
Oct 30 04:22:26.768327 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:26.768291    4877 generic.go:356] "Generic (PLEG): container finished" podID="62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" containerID="b210a18b4085f91c99c4a00560217f571c275805ed949b4ef77c6f69ee5e2853" exitCode=0
Oct 30 04:22:26.768793 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:26.768380    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv" event={"ID":"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c","Type":"ContainerDied","Data":"b210a18b4085f91c99c4a00560217f571c275805ed949b4ef77c6f69ee5e2853"}
Oct 30 04:22:27.926457 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:27.926416    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:28.023972 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.023938    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca\") pod \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") "
Oct 30 04:22:28.024186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.024027    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca\") pod \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") "
Oct 30 04:22:28.024186 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.024151    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l\" (UniqueName: \"kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l\") pod \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\" (UID: \"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c\") "
Oct 30 04:22:28.024369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.024346    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca" pod "62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" (UID: "62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:22:28.024514 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.024498    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:28.026106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.026079    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca" pod "62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" (UID: "62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:22:28.026240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.026216    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l" pod "62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" (UID: "62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c"). InnerVolumeSpecName "kube-api-access-hn28l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:22:28.125813 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.125775    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:28.125813 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.125808    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn28l\" (UniqueName: \"kubernetes.io/projected/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c-kube-api-access-hn28l\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:28.780728 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.780694    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"
Oct 30 04:22:28.780932 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.780738    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv" event={"ID":"62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c","Type":"ContainerDied","Data":"1c5bda3c6b647a11e81b3a23b651f406dbcb3cb202e225e8e3d32858d8842745"}
Oct 30 04:22:28.780932 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:28.780802    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5bda3c6b647a11e81b3a23b651f406dbcb3cb202e225e8e3d32858d8842745"
Oct 30 04:22:37.027138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:37.027102    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"]
Oct 30 04:22:37.036671 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:37.036642    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bxtgj"]
Oct 30 04:22:37.699490 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:37.699448    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57" path="/var/lib/kubelet/pods/404e6fbd-d2a6-4ed1-93e6-a3907eaa7f57/volumes"
Oct 30 04:22:41.344177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.344134    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"]
Oct 30 04:22:41.344790 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.344766    4877 cpu_manager.go:414] "containerMap: removing container" podUID="62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" containerName="revoke-keys"
Oct 30 04:22:41.344879 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.344791    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" containerName="revoke-keys"
Oct 30 04:22:41.344974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.344952    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" containerName="revoke-keys"
Oct 30 04:22:41.348399 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.348378    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.362432 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.362401    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:22:41.372678 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.372629    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:22:41.376271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.376238    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrm4x\" (UniqueName: \"kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.376450 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.376345    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.376528 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.376468    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.377393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.377365    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"]
Oct 30 04:22:41.477851 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.477816    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.478049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.477872    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrm4x\" (UniqueName: \"kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.478049 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.477920    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.478680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.478653    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.480344 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.480322    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.495827 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.495794    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrm4x\" (UniqueName: \"kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x\") pod \"maas-api-revoke-sub-a4aef7d7-z8zzp\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.662310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.662219    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:41.962411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:41.962381    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"]
Oct 30 04:22:41.965457 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:22:41.965409    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0878fcf1_a7de_44e5_a297_38fee37dac68.slice/crio-2a34669957dc33020e64734e0bd1311e0678f799f55639149c0ed3cc4dbd4c88 WatchSource:0}: Error finding container 2a34669957dc33020e64734e0bd1311e0678f799f55639149c0ed3cc4dbd4c88: Status 404 returned error can't find the container with id 2a34669957dc33020e64734e0bd1311e0678f799f55639149c0ed3cc4dbd4c88
Oct 30 04:22:42.880122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:42.880073    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp" event={"ID":"0878fcf1-a7de-44e5-a297-38fee37dac68","Type":"ContainerStarted","Data":"ed7b309ef0eb4c7de5a659cdb6bff65c606625d0dc146716ed40ee345862a5fa"}
Oct 30 04:22:42.880623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:42.880140    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp" event={"ID":"0878fcf1-a7de-44e5-a297-38fee37dac68","Type":"ContainerStarted","Data":"2a34669957dc33020e64734e0bd1311e0678f799f55639149c0ed3cc4dbd4c88"}
Oct 30 04:22:42.915743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:42.915696    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp" podStartSLOduration=1.9156837580000001 podStartE2EDuration="1.915683758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:42.914505088 +0000 UTC m=+4791.343394524" watchObservedRunningTime="2026-10-30 04:22:42.915683758 +0000 UTC m=+4791.344573176"
Oct 30 04:22:43.887868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:43.887829    4877 generic.go:356] "Generic (PLEG): container finished" podID="0878fcf1-a7de-44e5-a297-38fee37dac68" containerID="ed7b309ef0eb4c7de5a659cdb6bff65c606625d0dc146716ed40ee345862a5fa" exitCode=0
Oct 30 04:22:43.888291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:43.887911    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp" event={"ID":"0878fcf1-a7de-44e5-a297-38fee37dac68","Type":"ContainerDied","Data":"ed7b309ef0eb4c7de5a659cdb6bff65c606625d0dc146716ed40ee345862a5fa"}
Oct 30 04:22:45.055341 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.055318    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:45.116955 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.116917    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca\") pod \"0878fcf1-a7de-44e5-a297-38fee37dac68\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") "
Oct 30 04:22:45.117178 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.117050    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca\") pod \"0878fcf1-a7de-44e5-a297-38fee37dac68\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") "
Oct 30 04:22:45.117178 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.117130    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x\" (UniqueName: \"kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x\") pod \"0878fcf1-a7de-44e5-a297-38fee37dac68\" (UID: \"0878fcf1-a7de-44e5-a297-38fee37dac68\") "
Oct 30 04:22:45.117326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.117265    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca" pod "0878fcf1-a7de-44e5-a297-38fee37dac68" (UID: "0878fcf1-a7de-44e5-a297-38fee37dac68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:22:45.117492 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.117467    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:45.119882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.119856    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca" pod "0878fcf1-a7de-44e5-a297-38fee37dac68" (UID: "0878fcf1-a7de-44e5-a297-38fee37dac68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:22:45.119882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.119879    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x" pod "0878fcf1-a7de-44e5-a297-38fee37dac68" (UID: "0878fcf1-a7de-44e5-a297-38fee37dac68"). InnerVolumeSpecName "kube-api-access-zrm4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:22:45.218652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.218569    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0878fcf1-a7de-44e5-a297-38fee37dac68-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:45.218652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.218599    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrm4x\" (UniqueName: \"kubernetes.io/projected/0878fcf1-a7de-44e5-a297-38fee37dac68-kube-api-access-zrm4x\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:22:45.901468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.901433    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp" event={"ID":"0878fcf1-a7de-44e5-a297-38fee37dac68","Type":"ContainerDied","Data":"2a34669957dc33020e64734e0bd1311e0678f799f55639149c0ed3cc4dbd4c88"}
Oct 30 04:22:45.901691 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.901474    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"
Oct 30 04:22:45.901691 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:45.901485    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a34669957dc33020e64734e0bd1311e0678f799f55639149c0ed3cc4dbd4c88"
Oct 30 04:22:46.228496 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:22:46.228461    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:22:53.180147 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:53.180115    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:22:53.203653 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:53.203627    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:22:55.026450 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:55.026388    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"]
Oct 30 04:22:55.035375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:55.035346    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396e7f24-qmlmg"]
Oct 30 04:22:55.696754 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:55.696719    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bd6584-7335-4f53-8753-6eec8a51d775" path="/var/lib/kubelet/pods/a0bd6584-7335-4f53-8753-6eec8a51d775/volumes"
Oct 30 04:22:58.209696 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.209660    4877 scope.go:122] "RemoveContainer" containerID="7f70d681f344460a8efdd42593e2ee0ba44ff9b54cf134c2cbbe919036c028ba"
Oct 30 04:22:58.420080 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.420045    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"]
Oct 30 04:22:58.421018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.420987    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0878fcf1-a7de-44e5-a297-38fee37dac68" containerName="revoke-keys"
Oct 30 04:22:58.421018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.421016    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0878fcf1-a7de-44e5-a297-38fee37dac68" containerName="revoke-keys"
Oct 30 04:22:58.421328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.421133    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0878fcf1-a7de-44e5-a297-38fee37dac68" containerName="revoke-keys"
Oct 30 04:22:58.426702 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.426672    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.450090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.450010    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:22:58.450090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.450057    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:22:58.487207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.486828    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"]
Oct 30 04:22:58.557656 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.557618    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.557992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.557959    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dcm\" (UniqueName: \"kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.558551 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.558528    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.659751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.659718    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.659993 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.659979    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dcm\" (UniqueName: \"kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.660125 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.660111    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.661036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.660994    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.662521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.662490    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.698177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.698140    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dcm\" (UniqueName: \"kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm\") pod \"maas-api-revoke-sub-79805ca9-8982t\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:58.751094 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:58.750676    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:22:59.363279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:22:59.360211    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"]
Oct 30 04:22:59.365233 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:22:59.365185    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d608f18_adfc_4d76_a360_56df17b3d764.slice/crio-a20207b33eec61884affe4610abf5fb9bedd583de7aa3213624b52961e4e32d7 WatchSource:0}: Error finding container a20207b33eec61884affe4610abf5fb9bedd583de7aa3213624b52961e4e32d7: Status 404 returned error can't find the container with id a20207b33eec61884affe4610abf5fb9bedd583de7aa3213624b52961e4e32d7
Oct 30 04:23:00.082409 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:00.082367    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t" event={"ID":"6d608f18-adfc-4d76-a360-56df17b3d764","Type":"ContainerStarted","Data":"974ddcd49cf89b8d17ecebb705a342f5249797da5e074edb5a14229ff0cfaa13"}
Oct 30 04:23:00.082832 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:00.082806    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t" event={"ID":"6d608f18-adfc-4d76-a360-56df17b3d764","Type":"ContainerStarted","Data":"a20207b33eec61884affe4610abf5fb9bedd583de7aa3213624b52961e4e32d7"}
Oct 30 04:23:00.124446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:00.123610    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t" podStartSLOduration=2.123570641 podStartE2EDuration="2.123570641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:00.116991929 +0000 UTC m=+4808.545881348" watchObservedRunningTime="2026-10-30 04:23:00.123570641 +0000 UTC m=+4808.552460068"
Oct 30 04:23:01.089369 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:01.089331    4877 generic.go:356] "Generic (PLEG): container finished" podID="6d608f18-adfc-4d76-a360-56df17b3d764" containerID="974ddcd49cf89b8d17ecebb705a342f5249797da5e074edb5a14229ff0cfaa13" exitCode=0
Oct 30 04:23:01.089825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:01.089403    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t" event={"ID":"6d608f18-adfc-4d76-a360-56df17b3d764","Type":"ContainerDied","Data":"974ddcd49cf89b8d17ecebb705a342f5249797da5e074edb5a14229ff0cfaa13"}
Oct 30 04:23:02.242357 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.242331    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:23:02.320148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.320115    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca\") pod \"6d608f18-adfc-4d76-a360-56df17b3d764\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") "
Oct 30 04:23:02.320370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.320188    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca\") pod \"6d608f18-adfc-4d76-a360-56df17b3d764\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") "
Oct 30 04:23:02.320370 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.320245    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm\" (UniqueName: \"kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm\") pod \"6d608f18-adfc-4d76-a360-56df17b3d764\" (UID: \"6d608f18-adfc-4d76-a360-56df17b3d764\") "
Oct 30 04:23:02.320601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.320571    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca" pod "6d608f18-adfc-4d76-a360-56df17b3d764" (UID: "6d608f18-adfc-4d76-a360-56df17b3d764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:02.322468 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.322435    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca" pod "6d608f18-adfc-4d76-a360-56df17b3d764" (UID: "6d608f18-adfc-4d76-a360-56df17b3d764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:23:02.322608 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.322524    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm" pod "6d608f18-adfc-4d76-a360-56df17b3d764" (UID: "6d608f18-adfc-4d76-a360-56df17b3d764"). InnerVolumeSpecName "kube-api-access-d5dcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:02.421360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.421261    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:02.421360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.421303    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d608f18-adfc-4d76-a360-56df17b3d764-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:02.421360 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:02.421319    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dcm\" (UniqueName: \"kubernetes.io/projected/6d608f18-adfc-4d76-a360-56df17b3d764-kube-api-access-d5dcm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:03.103023 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:03.102995    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"
Oct 30 04:23:03.103224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:03.103036    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t" event={"ID":"6d608f18-adfc-4d76-a360-56df17b3d764","Type":"ContainerDied","Data":"a20207b33eec61884affe4610abf5fb9bedd583de7aa3213624b52961e4e32d7"}
Oct 30 04:23:03.103224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:03.103085    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20207b33eec61884affe4610abf5fb9bedd583de7aa3213624b52961e4e32d7"
Oct 30 04:23:12.226510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.225555    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"]
Oct 30 04:23:12.226510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.226343    4877 cpu_manager.go:414] "containerMap: removing container" podUID="6d608f18-adfc-4d76-a360-56df17b3d764" containerName="revoke-keys"
Oct 30 04:23:12.226510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.226365    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d608f18-adfc-4d76-a360-56df17b3d764" containerName="revoke-keys"
Oct 30 04:23:12.226510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.226483    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d608f18-adfc-4d76-a360-56df17b3d764" containerName="revoke-keys"
Oct 30 04:23:12.249177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.230230    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.249177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.245211    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"]
Oct 30 04:23:12.260638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.260597    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:23:12.260895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.260702    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:23:12.347343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.346977    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.347343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.347118    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwckp\" (UniqueName: \"kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.347343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.347173    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.448625 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.448589    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwckp\" (UniqueName: \"kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.448919 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.448901    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.449090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.449071    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.450016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.449992    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.459721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.459693    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.471163 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.471131    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwckp\" (UniqueName: \"kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp\") pod \"maas-api-revoke-sub-0b7e8edb-jb2f6\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:12.580631 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:12.580585    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:13.174888 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:13.174794    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"]
Oct 30 04:23:14.205935 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:14.205893    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6" event={"ID":"45176c6e-2969-4751-ad02-5a0775d12981","Type":"ContainerStarted","Data":"59127ccb0ff3bc42888ce9ee0fbe9cd25fdb0611f62992a909e48658d185f874"}
Oct 30 04:23:14.206350 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:14.205962    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6" event={"ID":"45176c6e-2969-4751-ad02-5a0775d12981","Type":"ContainerStarted","Data":"ef3e39e61dbc32a1fe32cb10a45ec7f9301a99df7b1a18dd16eef35818fb2037"}
Oct 30 04:23:14.240658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:14.240601    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6" podStartSLOduration=2.24058762 podStartE2EDuration="2.24058762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:14.239032369 +0000 UTC m=+4822.667921786" watchObservedRunningTime="2026-10-30 04:23:14.24058762 +0000 UTC m=+4822.669477037"
Oct 30 04:23:15.213682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:15.213644    4877 generic.go:356] "Generic (PLEG): container finished" podID="45176c6e-2969-4751-ad02-5a0775d12981" containerID="59127ccb0ff3bc42888ce9ee0fbe9cd25fdb0611f62992a909e48658d185f874" exitCode=0
Oct 30 04:23:15.214155 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:15.213714    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6" event={"ID":"45176c6e-2969-4751-ad02-5a0775d12981","Type":"ContainerDied","Data":"59127ccb0ff3bc42888ce9ee0fbe9cd25fdb0611f62992a909e48658d185f874"}
Oct 30 04:23:16.230754 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:23:16.230716    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:23:16.410525 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.410488    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:16.499777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.499686    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca\") pod \"45176c6e-2969-4751-ad02-5a0775d12981\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") "
Oct 30 04:23:16.499777 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.499742    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp\" (UniqueName: \"kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp\") pod \"45176c6e-2969-4751-ad02-5a0775d12981\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") "
Oct 30 04:23:16.500068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.499795    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca\") pod \"45176c6e-2969-4751-ad02-5a0775d12981\" (UID: \"45176c6e-2969-4751-ad02-5a0775d12981\") "
Oct 30 04:23:16.500123 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.500066    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca" pod "45176c6e-2969-4751-ad02-5a0775d12981" (UID: "45176c6e-2969-4751-ad02-5a0775d12981"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:16.502042 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.502016    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca" pod "45176c6e-2969-4751-ad02-5a0775d12981" (UID: "45176c6e-2969-4751-ad02-5a0775d12981"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:23:16.502169 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.502056    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp" pod "45176c6e-2969-4751-ad02-5a0775d12981" (UID: "45176c6e-2969-4751-ad02-5a0775d12981"). InnerVolumeSpecName "kube-api-access-rwckp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:16.600823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.600783    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:16.600823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.600813    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwckp\" (UniqueName: \"kubernetes.io/projected/45176c6e-2969-4751-ad02-5a0775d12981-kube-api-access-rwckp\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:16.600823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:16.600824    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45176c6e-2969-4751-ad02-5a0775d12981-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:17.227822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:17.227789    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"
Oct 30 04:23:17.227822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:17.227808    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6" event={"ID":"45176c6e-2969-4751-ad02-5a0775d12981","Type":"ContainerDied","Data":"ef3e39e61dbc32a1fe32cb10a45ec7f9301a99df7b1a18dd16eef35818fb2037"}
Oct 30 04:23:17.228052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:17.227858    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3e39e61dbc32a1fe32cb10a45ec7f9301a99df7b1a18dd16eef35818fb2037"
Oct 30 04:23:22.937126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.937085    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"]
Oct 30 04:23:22.937572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.937558    4877 cpu_manager.go:414] "containerMap: removing container" podUID="45176c6e-2969-4751-ad02-5a0775d12981" containerName="revoke-keys"
Oct 30 04:23:22.937620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.937573    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45176c6e-2969-4751-ad02-5a0775d12981" containerName="revoke-keys"
Oct 30 04:23:22.937664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.937656    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="45176c6e-2969-4751-ad02-5a0775d12981" containerName="revoke-keys"
Oct 30 04:23:22.941000 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.940983    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:22.958976 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.958948    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"]
Oct 30 04:23:22.968113 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.968079    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:22.968262 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.968127    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:22.968340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:22.968298    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjrq\" (UniqueName: \"kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.016889 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.016857    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:23:23.017092 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.017069    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:23:23.068836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.068800    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.069022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.068843    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.069022 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.068949    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjrq\" (UniqueName: \"kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.069662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.069638    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.071236 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.071214    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.085696 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.085670    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjrq\" (UniqueName: \"kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq\") pod \"maas-api-revoke-sub-3d4156f1-p276q\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.253882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.253802    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:23.552190 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:23.552154    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"]
Oct 30 04:23:23.553350 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:23:23.553322    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ca55d6_77e3_4c79_bf1f_142d0b198c0c.slice/crio-e010478dfa8fea5c956987565c7ce512c637c266c757d4aebefe193e4bd64853 WatchSource:0}: Error finding container e010478dfa8fea5c956987565c7ce512c637c266c757d4aebefe193e4bd64853: Status 404 returned error can't find the container with id e010478dfa8fea5c956987565c7ce512c637c266c757d4aebefe193e4bd64853
Oct 30 04:23:24.275075 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:24.275034    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q" event={"ID":"86ca55d6-77e3-4c79-bf1f-142d0b198c0c","Type":"ContainerStarted","Data":"480f20e4b61a5f06cbafc82f12288a4bc2c0973f1c3095f18ab0f42475485e1f"}
Oct 30 04:23:24.275485 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:24.275097    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q" event={"ID":"86ca55d6-77e3-4c79-bf1f-142d0b198c0c","Type":"ContainerStarted","Data":"e010478dfa8fea5c956987565c7ce512c637c266c757d4aebefe193e4bd64853"}
Oct 30 04:23:25.282859 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:25.282810    4877 generic.go:356] "Generic (PLEG): container finished" podID="86ca55d6-77e3-4c79-bf1f-142d0b198c0c" containerID="480f20e4b61a5f06cbafc82f12288a4bc2c0973f1c3095f18ab0f42475485e1f" exitCode=0
Oct 30 04:23:25.283293 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:25.282893    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q" event={"ID":"86ca55d6-77e3-4c79-bf1f-142d0b198c0c","Type":"ContainerDied","Data":"480f20e4b61a5f06cbafc82f12288a4bc2c0973f1c3095f18ab0f42475485e1f"}
Oct 30 04:23:26.438320 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.438297    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:26.501219 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.501186    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca\") pod \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") "
Oct 30 04:23:26.501436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.501356    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca\") pod \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") "
Oct 30 04:23:26.501436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.501393    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq\" (UniqueName: \"kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq\") pod \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\" (UID: \"86ca55d6-77e3-4c79-bf1f-142d0b198c0c\") "
Oct 30 04:23:26.501852 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.501813    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca" pod "86ca55d6-77e3-4c79-bf1f-142d0b198c0c" (UID: "86ca55d6-77e3-4c79-bf1f-142d0b198c0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:26.503470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.503441    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca" pod "86ca55d6-77e3-4c79-bf1f-142d0b198c0c" (UID: "86ca55d6-77e3-4c79-bf1f-142d0b198c0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:23:26.503594 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.503541    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq" pod "86ca55d6-77e3-4c79-bf1f-142d0b198c0c" (UID: "86ca55d6-77e3-4c79-bf1f-142d0b198c0c"). InnerVolumeSpecName "kube-api-access-lfjrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:26.603312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.603224    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:26.603312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.603258    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjrq\" (UniqueName: \"kubernetes.io/projected/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-kube-api-access-lfjrq\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:26.603312 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:26.603268    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ca55d6-77e3-4c79-bf1f-142d0b198c0c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:27.295870 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:27.295836    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"
Oct 30 04:23:27.296052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:27.295868    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q" event={"ID":"86ca55d6-77e3-4c79-bf1f-142d0b198c0c","Type":"ContainerDied","Data":"e010478dfa8fea5c956987565c7ce512c637c266c757d4aebefe193e4bd64853"}
Oct 30 04:23:27.296052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:27.295917    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e010478dfa8fea5c956987565c7ce512c637c266c757d4aebefe193e4bd64853"
Oct 30 04:23:40.689825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.689788    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"]
Oct 30 04:23:40.699105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.699079    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p276q"]
Oct 30 04:23:40.730386 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.730353    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"]
Oct 30 04:23:40.731088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.731061    4877 cpu_manager.go:414] "containerMap: removing container" podUID="86ca55d6-77e3-4c79-bf1f-142d0b198c0c" containerName="revoke-keys"
Oct 30 04:23:40.731088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.731083    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ca55d6-77e3-4c79-bf1f-142d0b198c0c" containerName="revoke-keys"
Oct 30 04:23:40.731292 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.731194    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ca55d6-77e3-4c79-bf1f-142d0b198c0c" containerName="revoke-keys"
Oct 30 04:23:40.734931 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.734906    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.748088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.748059    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:23:40.748228 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.748138    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:23:40.752617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.751361    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"]
Oct 30 04:23:40.791488 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.791453    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.791656 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.791518    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdz55\" (UniqueName: \"kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.791708 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.791667    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.893088 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.893056    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.893274 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.893219    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.893328 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.893281    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdz55\" (UniqueName: \"kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.893768 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.893748    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.895470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.895453    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:40.909944 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:40.909914    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdz55\" (UniqueName: \"kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55\") pod \"maas-api-revoke-sub-3d4156f1-jfxx8\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:41.051898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:41.051864    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:41.347258 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:41.347230    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"]
Oct 30 04:23:41.348464 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:23:41.348405    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c9be4d_1187_434e_a4f3_1385856a0c8c.slice/crio-2a6d92b5e935d1aaabe261663a6c06279435d7051b0f0f668a44b45212dff8e3 WatchSource:0}: Error finding container 2a6d92b5e935d1aaabe261663a6c06279435d7051b0f0f668a44b45212dff8e3: Status 404 returned error can't find the container with id 2a6d92b5e935d1aaabe261663a6c06279435d7051b0f0f668a44b45212dff8e3
Oct 30 04:23:41.413501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:41.413464    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8" event={"ID":"f9c9be4d-1187-434e-a4f3-1385856a0c8c","Type":"ContainerStarted","Data":"2a6d92b5e935d1aaabe261663a6c06279435d7051b0f0f668a44b45212dff8e3"}
Oct 30 04:23:41.705521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:41.705485    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ca55d6-77e3-4c79-bf1f-142d0b198c0c" path="/var/lib/kubelet/pods/86ca55d6-77e3-4c79-bf1f-142d0b198c0c/volumes"
Oct 30 04:23:42.421962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:42.421925    4877 generic.go:356] "Generic (PLEG): container finished" podID="f9c9be4d-1187-434e-a4f3-1385856a0c8c" containerID="99bf71ae10a3d3e1681d1a9c29399df1cc0d15f77709adef4e606847688de805" exitCode=0
Oct 30 04:23:42.422158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:42.421977    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8" event={"ID":"f9c9be4d-1187-434e-a4f3-1385856a0c8c","Type":"ContainerDied","Data":"99bf71ae10a3d3e1681d1a9c29399df1cc0d15f77709adef4e606847688de805"}
Oct 30 04:23:43.581869 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.581837    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:43.722463 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.722344    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca\") pod \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") "
Oct 30 04:23:43.722640 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.722466    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55\" (UniqueName: \"kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55\") pod \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") "
Oct 30 04:23:43.722640 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.722586    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca\") pod \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\" (UID: \"f9c9be4d-1187-434e-a4f3-1385856a0c8c\") "
Oct 30 04:23:43.723010 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.722980    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca" pod "f9c9be4d-1187-434e-a4f3-1385856a0c8c" (UID: "f9c9be4d-1187-434e-a4f3-1385856a0c8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:23:43.724500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.724474    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca" pod "f9c9be4d-1187-434e-a4f3-1385856a0c8c" (UID: "f9c9be4d-1187-434e-a4f3-1385856a0c8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:23:43.724604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.724534    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55" pod "f9c9be4d-1187-434e-a4f3-1385856a0c8c" (UID: "f9c9be4d-1187-434e-a4f3-1385856a0c8c"). InnerVolumeSpecName "kube-api-access-kdz55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:23:43.824151 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.824108    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:43.824151 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.824153    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9be4d-1187-434e-a4f3-1385856a0c8c-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:43.824414 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:43.824167    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdz55\" (UniqueName: \"kubernetes.io/projected/f9c9be4d-1187-434e-a4f3-1385856a0c8c-kube-api-access-kdz55\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:23:44.437404 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:44.437366    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"
Oct 30 04:23:44.437404 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:44.437379    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8" event={"ID":"f9c9be4d-1187-434e-a4f3-1385856a0c8c","Type":"ContainerDied","Data":"2a6d92b5e935d1aaabe261663a6c06279435d7051b0f0f668a44b45212dff8e3"}
Oct 30 04:23:44.437644 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:44.437446    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6d92b5e935d1aaabe261663a6c06279435d7051b0f0f668a44b45212dff8e3"
Oct 30 04:23:46.229118 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:23:46.229082    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:23:58.238304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:58.238265    4877 scope.go:122] "RemoveContainer" containerID="9eef6905cd4ea2c5a918ee0b34a34ab9076f40e7dec695bd5f1646ad6ff698c6"
Oct 30 04:23:58.248669 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:58.248648    4877 scope.go:122] "RemoveContainer" containerID="fc2c04081ec0d6e3cdb2a96859f2642b18abafc1b2f1c340bb2c2544771708fc"
Oct 30 04:23:58.257514 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:23:58.257488    4877 scope.go:122] "RemoveContainer" containerID="88833fde12df2353fc5c487263233a046e85433921fa703c140c6e99d36a25ec"
Oct 30 04:24:06.318565 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:06.318524    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:24:06.319199 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:06.319153    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:24:10.434742 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.434711    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"]
Oct 30 04:24:10.435154 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.435132    4877 cpu_manager.go:414] "containerMap: removing container" podUID="f9c9be4d-1187-434e-a4f3-1385856a0c8c" containerName="revoke-keys"
Oct 30 04:24:10.435154 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.435143    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c9be4d-1187-434e-a4f3-1385856a0c8c" containerName="revoke-keys"
Oct 30 04:24:10.435249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.435238    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c9be4d-1187-434e-a4f3-1385856a0c8c" containerName="revoke-keys"
Oct 30 04:24:10.440026 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.440004    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.457349 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.457311    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"]
Oct 30 04:24:10.463286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.463260    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:24:10.463521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.463328    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:24:10.519856 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.519823    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dq5h\" (UniqueName: \"kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.520036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.519877    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.520036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.519979    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.620850 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.620818    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dq5h\" (UniqueName: \"kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.621087 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.620863    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.621087 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.620910    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.621704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.621682    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.623410 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.623385    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.638634 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.638610    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dq5h\" (UniqueName: \"kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h\") pod \"maas-api-revoke-sub-e2fc9c5b-g644n\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:10.753342 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:10.753262    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:11.050874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:11.050840    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"]
Oct 30 04:24:11.052884 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:24:11.052855    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eea2de9_d4ff_41de_a861_afcaef18c9bf.slice/crio-a57d3b8502ce106cd6c2fe575f0df10ec96814a8ba6ad37173df3730fc9b0ee3 WatchSource:0}: Error finding container a57d3b8502ce106cd6c2fe575f0df10ec96814a8ba6ad37173df3730fc9b0ee3: Status 404 returned error can't find the container with id a57d3b8502ce106cd6c2fe575f0df10ec96814a8ba6ad37173df3730fc9b0ee3
Oct 30 04:24:11.621391 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:11.621346    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n" event={"ID":"0eea2de9-d4ff-41de-a861-afcaef18c9bf","Type":"ContainerStarted","Data":"3bb34604429dff7e814a0ed0fe0f11480d88cd82c498d11c66f7d46de69f2373"}
Oct 30 04:24:11.621918 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:11.621408    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n" event={"ID":"0eea2de9-d4ff-41de-a861-afcaef18c9bf","Type":"ContainerStarted","Data":"a57d3b8502ce106cd6c2fe575f0df10ec96814a8ba6ad37173df3730fc9b0ee3"}
Oct 30 04:24:11.658047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:11.657949    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n" podStartSLOduration=1.657936396 podStartE2EDuration="1.657936396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:11.654550409 +0000 UTC m=+4880.083439826" watchObservedRunningTime="2026-10-30 04:24:11.657936396 +0000 UTC m=+4880.086825813"
Oct 30 04:24:12.629329 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:12.629292    4877 generic.go:356] "Generic (PLEG): container finished" podID="0eea2de9-d4ff-41de-a861-afcaef18c9bf" containerID="3bb34604429dff7e814a0ed0fe0f11480d88cd82c498d11c66f7d46de69f2373" exitCode=0
Oct 30 04:24:12.629772 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:12.629385    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n" event={"ID":"0eea2de9-d4ff-41de-a861-afcaef18c9bf","Type":"ContainerDied","Data":"3bb34604429dff7e814a0ed0fe0f11480d88cd82c498d11c66f7d46de69f2373"}
Oct 30 04:24:13.775286 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.775258    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:13.852832 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.852795    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca\") pod \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") "
Oct 30 04:24:13.853029 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.852881    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca\") pod \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") "
Oct 30 04:24:13.853029 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.852927    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h\" (UniqueName: \"kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h\") pod \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\" (UID: \"0eea2de9-d4ff-41de-a861-afcaef18c9bf\") "
Oct 30 04:24:13.853246 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.853214    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca" pod "0eea2de9-d4ff-41de-a861-afcaef18c9bf" (UID: "0eea2de9-d4ff-41de-a861-afcaef18c9bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:24:13.854984 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.854957    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca" pod "0eea2de9-d4ff-41de-a861-afcaef18c9bf" (UID: "0eea2de9-d4ff-41de-a861-afcaef18c9bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:24:13.855085 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.855035    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h" pod "0eea2de9-d4ff-41de-a861-afcaef18c9bf" (UID: "0eea2de9-d4ff-41de-a861-afcaef18c9bf"). InnerVolumeSpecName "kube-api-access-7dq5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:24:13.953979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.953873    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:13.953979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.953923    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eea2de9-d4ff-41de-a861-afcaef18c9bf-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:13.953979 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:13.953933    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dq5h\" (UniqueName: \"kubernetes.io/projected/0eea2de9-d4ff-41de-a861-afcaef18c9bf-kube-api-access-7dq5h\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:14.643326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:14.643289    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n" event={"ID":"0eea2de9-d4ff-41de-a861-afcaef18c9bf","Type":"ContainerDied","Data":"a57d3b8502ce106cd6c2fe575f0df10ec96814a8ba6ad37173df3730fc9b0ee3"}
Oct 30 04:24:14.643326 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:14.643333    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57d3b8502ce106cd6c2fe575f0df10ec96814a8ba6ad37173df3730fc9b0ee3"
Oct 30 04:24:14.643680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:14.643331    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"
Oct 30 04:24:16.227719 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:24:16.227675    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:24:20.774313 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.774268    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"]
Oct 30 04:24:20.779414 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.779378    4877 cpu_manager.go:414] "containerMap: removing container" podUID="0eea2de9-d4ff-41de-a861-afcaef18c9bf" containerName="revoke-keys"
Oct 30 04:24:20.779414 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.779413    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eea2de9-d4ff-41de-a861-afcaef18c9bf" containerName="revoke-keys"
Oct 30 04:24:20.779648 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.779618    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eea2de9-d4ff-41de-a861-afcaef18c9bf" containerName="revoke-keys"
Oct 30 04:24:20.783584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.783566    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:20.791572 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.791307    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"]
Oct 30 04:24:20.797775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.797750    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:24:20.797917 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.797832    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:24:20.922115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.922079    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtcx\" (UniqueName: \"kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:20.922336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.922143    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:20.922336 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:20.922295    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.023105 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.023073    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.023322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.023134    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtcx\" (UniqueName: \"kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.023322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.023190    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.023914 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.023890    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.025684 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.025626    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.040191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.040168    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtcx\" (UniqueName: \"kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx\") pod \"maas-api-revoke-sub-5a29efb3-t74s8\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.100238 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.100209    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:21.404898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.404859    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"]
Oct 30 04:24:21.405770 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:24:21.405741    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9709babe_5874_4d63_9bdb_dd7548127c7f.slice/crio-7524db5183ddb12b38373fc710acd2a5080e96dcb815082bd10355339f6e92a8 WatchSource:0}: Error finding container 7524db5183ddb12b38373fc710acd2a5080e96dcb815082bd10355339f6e92a8: Status 404 returned error can't find the container with id 7524db5183ddb12b38373fc710acd2a5080e96dcb815082bd10355339f6e92a8
Oct 30 04:24:21.699340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:21.699309    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8" event={"ID":"9709babe-5874-4d63-9bdb-dd7548127c7f","Type":"ContainerStarted","Data":"7524db5183ddb12b38373fc710acd2a5080e96dcb815082bd10355339f6e92a8"}
Oct 30 04:24:22.705299 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:22.705256    4877 generic.go:356] "Generic (PLEG): container finished" podID="9709babe-5874-4d63-9bdb-dd7548127c7f" containerID="08cdad42f6c6e1d71196b4ab8dad9c707fcb2c7e1e2a1f06accb42c5288c3b75" exitCode=0
Oct 30 04:24:22.705863 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:22.705342    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8" event={"ID":"9709babe-5874-4d63-9bdb-dd7548127c7f","Type":"ContainerDied","Data":"08cdad42f6c6e1d71196b4ab8dad9c707fcb2c7e1e2a1f06accb42c5288c3b75"}
Oct 30 04:24:24.186263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.186233    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:24.305791 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.305698    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca\") pod \"9709babe-5874-4d63-9bdb-dd7548127c7f\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") "
Oct 30 04:24:24.306130 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.306101    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx\" (UniqueName: \"kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx\") pod \"9709babe-5874-4d63-9bdb-dd7548127c7f\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") "
Oct 30 04:24:24.307499 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.307475    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca\") pod \"9709babe-5874-4d63-9bdb-dd7548127c7f\" (UID: \"9709babe-5874-4d63-9bdb-dd7548127c7f\") "
Oct 30 04:24:24.308334 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.308311    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca" pod "9709babe-5874-4d63-9bdb-dd7548127c7f" (UID: "9709babe-5874-4d63-9bdb-dd7548127c7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:24:24.319452 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.318992    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx" pod "9709babe-5874-4d63-9bdb-dd7548127c7f" (UID: "9709babe-5874-4d63-9bdb-dd7548127c7f"). InnerVolumeSpecName "kube-api-access-ljtcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:24:24.352445 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.344986    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca" pod "9709babe-5874-4d63-9bdb-dd7548127c7f" (UID: "9709babe-5874-4d63-9bdb-dd7548127c7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:24:24.425447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.424916    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:24.425447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.424967    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9709babe-5874-4d63-9bdb-dd7548127c7f-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:24.425447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.424986    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtcx\" (UniqueName: \"kubernetes.io/projected/9709babe-5874-4d63-9bdb-dd7548127c7f-kube-api-access-ljtcx\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:24.743511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.740435    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"
Oct 30 04:24:24.743511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.740466    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8" event={"ID":"9709babe-5874-4d63-9bdb-dd7548127c7f","Type":"ContainerDied","Data":"7524db5183ddb12b38373fc710acd2a5080e96dcb815082bd10355339f6e92a8"}
Oct 30 04:24:24.743511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:24.740528    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7524db5183ddb12b38373fc710acd2a5080e96dcb815082bd10355339f6e92a8"
Oct 30 04:24:37.248431 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.248337    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"]
Oct 30 04:24:37.257860 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.257831    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jfxx8"]
Oct 30 04:24:37.289821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.289787    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"]
Oct 30 04:24:37.290274 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.290257    4877 cpu_manager.go:414] "containerMap: removing container" podUID="9709babe-5874-4d63-9bdb-dd7548127c7f" containerName="revoke-keys"
Oct 30 04:24:37.290274 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.290271    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9709babe-5874-4d63-9bdb-dd7548127c7f" containerName="revoke-keys"
Oct 30 04:24:37.290486 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.290373    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="9709babe-5874-4d63-9bdb-dd7548127c7f" containerName="revoke-keys"
Oct 30 04:24:37.293836 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.293814    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.306261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.306236    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:24:37.306261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.306234    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:24:37.313806 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.313778    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"]
Oct 30 04:24:37.363682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.363646    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.363682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.363682    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.363994 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.363855    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq46m\" (UniqueName: \"kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.464970 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.464926    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.464970 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.464975    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.465193 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.465056    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq46m\" (UniqueName: \"kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.465694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.465674    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.467354 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.467331    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.482227 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.482189    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq46m\" (UniqueName: \"kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m\") pod \"maas-api-revoke-sub-3d4156f1-9kxpf\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.607922 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.607877    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:37.699769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.699732    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c9be4d-1187-434e-a4f3-1385856a0c8c" path="/var/lib/kubelet/pods/f9c9be4d-1187-434e-a4f3-1385856a0c8c/volumes"
Oct 30 04:24:37.904241 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:37.904213    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"]
Oct 30 04:24:37.906234 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:24:37.906203    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f23594_9d0e_4faf_985b_dbfd67fd9ffd.slice/crio-28dfb8ee49ce42c6726ef6c8c0d2babe1ed032a764f588128403ca6a3d8e2c62 WatchSource:0}: Error finding container 28dfb8ee49ce42c6726ef6c8c0d2babe1ed032a764f588128403ca6a3d8e2c62: Status 404 returned error can't find the container with id 28dfb8ee49ce42c6726ef6c8c0d2babe1ed032a764f588128403ca6a3d8e2c62
Oct 30 04:24:38.823613 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:38.823578    4877 generic.go:356] "Generic (PLEG): container finished" podID="54f23594-9d0e-4faf-985b-dbfd67fd9ffd" containerID="c41f0a4c0e12b18582612ab262d0f7a5da9b11786663991347e344453e9f1162" exitCode=0
Oct 30 04:24:38.824016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:38.823664    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf" event={"ID":"54f23594-9d0e-4faf-985b-dbfd67fd9ffd","Type":"ContainerDied","Data":"c41f0a4c0e12b18582612ab262d0f7a5da9b11786663991347e344453e9f1162"}
Oct 30 04:24:38.824016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:38.823717    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf" event={"ID":"54f23594-9d0e-4faf-985b-dbfd67fd9ffd","Type":"ContainerStarted","Data":"28dfb8ee49ce42c6726ef6c8c0d2babe1ed032a764f588128403ca6a3d8e2c62"}
Oct 30 04:24:39.979090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:39.979056    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:39.987405 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:39.987378    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca\") pod \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") "
Oct 30 04:24:39.987548 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:39.987494    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca\") pod \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") "
Oct 30 04:24:39.987844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:39.987822    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca" pod "54f23594-9d0e-4faf-985b-dbfd67fd9ffd" (UID: "54f23594-9d0e-4faf-985b-dbfd67fd9ffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:24:39.989441 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:39.989409    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca" pod "54f23594-9d0e-4faf-985b-dbfd67fd9ffd" (UID: "54f23594-9d0e-4faf-985b-dbfd67fd9ffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:24:40.088554 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.088468    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m\" (UniqueName: \"kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m\") pod \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\" (UID: \"54f23594-9d0e-4faf-985b-dbfd67fd9ffd\") "
Oct 30 04:24:40.088775 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.088761    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:40.088822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.088778    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:40.090552 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.090527    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m" pod "54f23594-9d0e-4faf-985b-dbfd67fd9ffd" (UID: "54f23594-9d0e-4faf-985b-dbfd67fd9ffd"). InnerVolumeSpecName "kube-api-access-tq46m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:24:40.189362 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.189318    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq46m\" (UniqueName: \"kubernetes.io/projected/54f23594-9d0e-4faf-985b-dbfd67fd9ffd-kube-api-access-tq46m\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:24:40.836916 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.836881    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"
Oct 30 04:24:40.837115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.836916    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf" event={"ID":"54f23594-9d0e-4faf-985b-dbfd67fd9ffd","Type":"ContainerDied","Data":"28dfb8ee49ce42c6726ef6c8c0d2babe1ed032a764f588128403ca6a3d8e2c62"}
Oct 30 04:24:40.837115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:40.836976    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28dfb8ee49ce42c6726ef6c8c0d2babe1ed032a764f588128403ca6a3d8e2c62"
Oct 30 04:24:46.228689 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:24:46.228648    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:24:51.271788 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:51.271730    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:24:51.272233 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:51.271812    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fs5rj" podUID="11ff9767-6324-495f-be0e-6bca50031925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:24:53.073680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.073641    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"]
Oct 30 04:24:53.074207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.074155    4877 cpu_manager.go:414] "containerMap: removing container" podUID="54f23594-9d0e-4faf-985b-dbfd67fd9ffd" containerName="revoke-keys"
Oct 30 04:24:53.074207 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.074169    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f23594-9d0e-4faf-985b-dbfd67fd9ffd" containerName="revoke-keys"
Oct 30 04:24:53.074306 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.074251    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f23594-9d0e-4faf-985b-dbfd67fd9ffd" containerName="revoke-keys"
Oct 30 04:24:53.077738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.077719    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.096070 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.096046    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a9dee1-openshift-default-dockercfg-9mtb8\""
Oct 30 04:24:53.127750 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.127709    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.127923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.127758    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.127923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.127857    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.127923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.127889    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.127923 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.127911    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.128119 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.128000    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.128119 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.128042    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.128119 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.128067    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.128224 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.128132    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8l66\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.204570 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.204528    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"]
Oct 30 04:24:53.229106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229070    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229106 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229111    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229152    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8l66\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229244    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229276    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229316    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229350    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229387    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229440    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229604    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229694 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229643    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229848    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.229987 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.229929    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.230470 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.230414    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.232226 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.232201    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.232504 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.232484    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.256602 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.256565    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.256769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.256715    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8l66\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66\") pod \"e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") " pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.392489 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.392371    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:53.726174 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.726070    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"]
Oct 30 04:24:53.730144 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.730092    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:53.730275 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.730233    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:53.730332 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.730315    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:24:53.958890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.958845    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" event={"ID":"bb2d08db-9637-4088-8d99-40745a164ca7","Type":"ContainerStarted","Data":"3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"}
Oct 30 04:24:53.959114 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:53.958907    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" event={"ID":"bb2d08db-9637-4088-8d99-40745a164ca7","Type":"ContainerStarted","Data":"f60169d0e1fb7ba46ba899ffbb5eec32f554530a37b133bf4facda94cd34df68"}
Oct 30 04:24:54.006159 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:54.006027    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podStartSLOduration=1.006007133 podStartE2EDuration="1.006007133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:53.995495616 +0000 UTC m=+4922.424385035" watchObservedRunningTime="2026-10-30 04:24:54.006007133 +0000 UTC m=+4922.434896552"
Oct 30 04:24:54.393858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:54.393822    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:54.398743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:54.398718    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:54.966172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:54.966134    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:54.967247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:54.967225    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:24:58.258256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.258225    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"]
Oct 30 04:24:58.262215 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.262195    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.281449 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.281390    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a9dee1\""
Oct 30 04:24:58.292703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.292673    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-515ae9de-dockercfg-bj9xl\""
Oct 30 04:24:58.298007 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.297975    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"]
Oct 30 04:24:58.353521 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.353480    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"]
Oct 30 04:24:58.359263 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.359231    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.373956 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.373924    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"]
Oct 30 04:24:58.389395 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.389356    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnln\" (UniqueName: \"kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln\") pod \"payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.389611 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.389501    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.490489 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.490418    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.490679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.490519    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.490725 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.490664    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsfz\" (UniqueName: \"kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz\") pod \"payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.490768 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.490745    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnln\" (UniqueName: \"kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln\") pod \"payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.491141 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.491123    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.508094 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.508052    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnln\" (UniqueName: \"kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln\") pod \"payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.577415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.577376    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:58.591895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.591855    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsfz\" (UniqueName: \"kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz\") pod \"payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.592034 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.591962    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.593012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.592984    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.610138 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.610096    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsfz\" (UniqueName: \"kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz\") pod \"payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.678240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.678205    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:24:58.884001 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:24:58.883957    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749c0169_3d79_4d0e_8a25_51ef53737676.slice/crio-b197165c9f274b89404e592b4afd59de0858ba7e4996eca3fd8fb7ec45ab97b4 WatchSource:0}: Error finding container b197165c9f274b89404e592b4afd59de0858ba7e4996eca3fd8fb7ec45ab97b4: Status 404 returned error can't find the container with id b197165c9f274b89404e592b4afd59de0858ba7e4996eca3fd8fb7ec45ab97b4
Oct 30 04:24:58.886781 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.886755    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"]
Oct 30 04:24:58.977042 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.976996    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"]
Oct 30 04:24:58.978146 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:24:58.978120    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda760e24a_c838_41f7_a3fd_92bed4f0108d.slice/crio-a230f603b85c0622725f1dbad346888393c43394ac42c997bc79f026c27edc7f WatchSource:0}: Error finding container a230f603b85c0622725f1dbad346888393c43394ac42c997bc79f026c27edc7f: Status 404 returned error can't find the container with id a230f603b85c0622725f1dbad346888393c43394ac42c997bc79f026c27edc7f
Oct 30 04:24:58.999012 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.998980    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" event={"ID":"749c0169-3d79-4d0e-8a25-51ef53737676","Type":"ContainerStarted","Data":"250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54"}
Oct 30 04:24:58.999102 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.999043    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" event={"ID":"749c0169-3d79-4d0e-8a25-51ef53737676","Type":"ContainerStarted","Data":"b197165c9f274b89404e592b4afd59de0858ba7e4996eca3fd8fb7ec45ab97b4"}
Oct 30 04:24:58.999196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:58.999183    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:24:59.000188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:59.000165    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" event={"ID":"a760e24a-c838-41f7-a3fd-92bed4f0108d","Type":"ContainerStarted","Data":"a230f603b85c0622725f1dbad346888393c43394ac42c997bc79f026c27edc7f"}
Oct 30 04:24:59.034188 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:24:59.034077    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" podStartSLOduration=1.034059915 podStartE2EDuration="1.034059915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:59.031770894 +0000 UTC m=+4927.460660313" watchObservedRunningTime="2026-10-30 04:24:59.034059915 +0000 UTC m=+4927.462949334"
Oct 30 04:25:00.008509 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:00.008467    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" event={"ID":"a760e24a-c838-41f7-a3fd-92bed4f0108d","Type":"ContainerStarted","Data":"f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8"}
Oct 30 04:25:00.008914 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:00.008540    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:25:00.044218 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:00.044075    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" podStartSLOduration=2.044057939 podStartE2EDuration="2.044057939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:00.041980456 +0000 UTC m=+4928.470869876" watchObservedRunningTime="2026-10-30 04:25:00.044057939 +0000 UTC m=+4928.472947357"
Oct 30 04:25:10.011539 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:10.011507    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:25:11.007608 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:11.007564    4877 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:25:11.007809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:11.007636    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:25:11.017895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:11.017870    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:25:12.089844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:12.089810    4877 generic.go:356] "Generic (PLEG): container finished" podID="bb2d08db-9637-4088-8d99-40745a164ca7" containerID="3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8" exitCode=0
Oct 30 04:25:12.090402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:12.089885    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" event={"ID":"bb2d08db-9637-4088-8d99-40745a164ca7","Type":"ContainerDied","Data":"3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"}
Oct 30 04:25:12.090402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:12.090307    4877 scope.go:122] "RemoveContainer" containerID="3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"
Oct 30 04:25:12.091536 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:12.091509    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:12.091604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:12.091579    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:12.091643 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:12.091612    4877 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 04:25:13.109403 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.109362    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" event={"ID":"bb2d08db-9637-4088-8d99-40745a164ca7","Type":"ContainerStarted","Data":"f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773"}
Oct 30 04:25:13.401858 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.401748    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:25:13.402068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.401877    4877 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body=
Oct 30 04:25:13.402068 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.401931    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused"
Oct 30 04:25:13.805561 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.801889    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"]
Oct 30 04:25:13.811574 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.811528    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:13.817073 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.817032    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"]
Oct 30 04:25:13.824165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.824138    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a9dee1\"/\"default-dockercfg-dhjgp\""
Oct 30 04:25:13.824322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.824154    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a9dee1\"/\"openshift-service-ca.crt\""
Oct 30 04:25:13.824322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.824160    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a9dee1\"/\"simulator-single-main-a9dee1-kserve-self-signed-certs\""
Oct 30 04:25:13.824322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.824161    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a9dee1\"/\"kube-root-ca.crt\""
Oct 30 04:25:13.900114 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.900064    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htcxf\" (UniqueName: \"kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:13.900618 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.900593    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:13.900881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.900863    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:13.901045 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.901016    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:13.901203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.901187    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:13.901329 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:13.901315    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003146 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003092    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003182    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003215    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003275    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003314    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003620 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003368    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htcxf\" (UniqueName: \"kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003670 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003616    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003670 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003644    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.003950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.003926    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.005849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.005817    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.006280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.006256    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.039040 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.023252    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htcxf\" (UniqueName: \"kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf\") pod \"simulator-single-main-a9dee1-kserve-6557459fc-v6dt4\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") " pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.129587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.129505    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:14.402952 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.402854    4877 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body=
Oct 30 04:25:14.402952 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.402925    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused"
Oct 30 04:25:14.449323 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:14.449288    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"]
Oct 30 04:25:14.450235 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:25:14.450204    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc364d2e_0ef0_40e6_b645_d93151d31179.slice/crio-f15ee71b7aaa90fa8da197dd927502b2cc89da163c0d127527606cb35612d5a7 WatchSource:0}: Error finding container f15ee71b7aaa90fa8da197dd927502b2cc89da163c0d127527606cb35612d5a7: Status 404 returned error can't find the container with id f15ee71b7aaa90fa8da197dd927502b2cc89da163c0d127527606cb35612d5a7
Oct 30 04:25:15.124529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:15.124489    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4" event={"ID":"bc364d2e-0ef0-40e6-b645-d93151d31179","Type":"ContainerStarted","Data":"56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702"}
Oct 30 04:25:15.124529 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:15.124533    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4" event={"ID":"bc364d2e-0ef0-40e6-b645-d93151d31179","Type":"ContainerStarted","Data":"f15ee71b7aaa90fa8da197dd927502b2cc89da163c0d127527606cb35612d5a7"}
Oct 30 04:25:15.129652 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:15.129623    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:15.158094 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:15.158036    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4" podStartSLOduration=2.158018539 podStartE2EDuration="2.158018539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:15.157717891 +0000 UTC m=+4943.586607309" watchObservedRunningTime="2026-10-30 04:25:15.158018539 +0000 UTC m=+4943.586907957"
Oct 30 04:25:15.393406 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:15.393311    4877 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body=
Oct 30 04:25:15.393723 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:15.393691    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused"
Oct 30 04:25:16.235363 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:25:16.235326    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:25:16.399546 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:16.397058    4877 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body=
Oct 30 04:25:16.399546 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:16.397118    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused"
Oct 30 04:25:17.401636 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:17.401592    4877 patch_prober.go:29] interesting pod/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused" start-of-body=
Oct 30 04:25:17.402271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:17.401652    4877 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.115:15021/healthz/ready\": dial tcp 10.133.1.115:15021: connect: connection refused"
Oct 30 04:25:18.614853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:18.614814    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:25:18.615377 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:18.615313    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:25:18.616137 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:18.616115    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:25:24.130558 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:24.130520    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:24.143209 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:24.143180    4877 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:24.198567 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:24.198535    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:25:33.024888 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:33.024848    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"]
Oct 30 04:25:33.034330 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:33.034301    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-mxj2w"]
Oct 30 04:25:33.697416 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:33.697383    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d2fc9c-865b-47c9-996b-1c627debbf8f" path="/var/lib/kubelet/pods/d0d2fc9c-865b-47c9-996b-1c627debbf8f/volumes"
Oct 30 04:25:45.831950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.831909    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"]
Oct 30 04:25:45.836843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.836815    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.840595 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.840564    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwpd\" (UniqueName: \"kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.840735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.840628    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.840735 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.840680    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.850962 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.850936    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"]
Oct 30 04:25:45.857247 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.857226    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:25:45.857453 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.857413    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:25:45.941350 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.941316    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwpd\" (UniqueName: \"kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.941587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.941372    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.941587 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.941391    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.942115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.942089    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.943761 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.943742    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:45.958232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:45.958203    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwpd\" (UniqueName: \"kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd\") pod \"maas-api-revoke-sub-ef440a36-9rvcj\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:46.152192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:46.152093    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:46.239299 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:25:46.239256    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:25:46.450545 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:46.450519    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"]
Oct 30 04:25:46.451904 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:25:46.451870    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ba4446_ca90_4760_a2a6_99150b82e360.slice/crio-a6705be27bd91572d68051c5e7f35d11029a0e8e7f7b8598435a65b34d2fbacd WatchSource:0}: Error finding container a6705be27bd91572d68051c5e7f35d11029a0e8e7f7b8598435a65b34d2fbacd: Status 404 returned error can't find the container with id a6705be27bd91572d68051c5e7f35d11029a0e8e7f7b8598435a65b34d2fbacd
Oct 30 04:25:47.350642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:47.350609    4877 generic.go:356] "Generic (PLEG): container finished" podID="92ba4446-ca90-4760-a2a6-99150b82e360" containerID="77ad9bb214b23211af2445d864ec73fb1e656d49e1908f0bc00942e848a00357" exitCode=0
Oct 30 04:25:47.351061 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:47.350690    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj" event={"ID":"92ba4446-ca90-4760-a2a6-99150b82e360","Type":"ContainerDied","Data":"77ad9bb214b23211af2445d864ec73fb1e656d49e1908f0bc00942e848a00357"}
Oct 30 04:25:47.351061 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:47.350744    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj" event={"ID":"92ba4446-ca90-4760-a2a6-99150b82e360","Type":"ContainerStarted","Data":"a6705be27bd91572d68051c5e7f35d11029a0e8e7f7b8598435a65b34d2fbacd"}
Oct 30 04:25:48.499617 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.499586    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:48.566948 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.566912    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca\") pod \"92ba4446-ca90-4760-a2a6-99150b82e360\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") "
Oct 30 04:25:48.567158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.567024    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca\") pod \"92ba4446-ca90-4760-a2a6-99150b82e360\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") "
Oct 30 04:25:48.567158 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.567077    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd\" (UniqueName: \"kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd\") pod \"92ba4446-ca90-4760-a2a6-99150b82e360\" (UID: \"92ba4446-ca90-4760-a2a6-99150b82e360\") "
Oct 30 04:25:48.567466 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.567435    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca" pod "92ba4446-ca90-4760-a2a6-99150b82e360" (UID: "92ba4446-ca90-4760-a2a6-99150b82e360"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:25:48.569744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.569720    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca" pod "92ba4446-ca90-4760-a2a6-99150b82e360" (UID: "92ba4446-ca90-4760-a2a6-99150b82e360"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:25:48.569744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.569733    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd" pod "92ba4446-ca90-4760-a2a6-99150b82e360" (UID: "92ba4446-ca90-4760-a2a6-99150b82e360"). InnerVolumeSpecName "kube-api-access-6rwpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:25:48.668732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.668649    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:25:48.668732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.668679    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ba4446-ca90-4760-a2a6-99150b82e360-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:25:48.668732 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:48.668692    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rwpd\" (UniqueName: \"kubernetes.io/projected/92ba4446-ca90-4760-a2a6-99150b82e360-kube-api-access-6rwpd\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:25:49.364862 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:49.364828    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"
Oct 30 04:25:49.364862 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:49.364848    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj" event={"ID":"92ba4446-ca90-4760-a2a6-99150b82e360","Type":"ContainerDied","Data":"a6705be27bd91572d68051c5e7f35d11029a0e8e7f7b8598435a65b34d2fbacd"}
Oct 30 04:25:49.365173 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:49.364936    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6705be27bd91572d68051c5e7f35d11029a0e8e7f7b8598435a65b34d2fbacd"
Oct 30 04:25:57.025890 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:57.025846    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"]
Oct 30 04:25:57.035534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:57.035500    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xbjnt"]
Oct 30 04:25:57.696893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:25:57.696860    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1712fa6a-3a6c-4bea-8db2-6b0278499014" path="/var/lib/kubelet/pods/1712fa6a-3a6c-4bea-8db2-6b0278499014/volumes"
Oct 30 04:26:01.788592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:01.788551    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"]
Oct 30 04:26:03.805630 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:03.805534    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"]
Oct 30 04:26:03.840517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:03.840473    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"]
Oct 30 04:26:08.805891 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:08.805843    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerName="payload-processing" containerID="cri-o://f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8" gracePeriod=25
Oct 30 04:26:08.841664 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:08.841618    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" containerName="payload-pre-processing" containerID="cri-o://250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54" gracePeriod=25
Oct 30 04:26:10.010769 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:10.010719    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:26:10.597773 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:10.597735    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"]
Oct 30 04:26:10.598280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:10.598245    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy" containerID="cri-o://f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773" gracePeriod=30
Oct 30 04:26:11.015658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.015572    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:26:11.840199 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.840161    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"]
Oct 30 04:26:11.849723 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.849694    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kxpf"]
Oct 30 04:26:11.882532 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.882504    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"]
Oct 30 04:26:11.882985 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.882971    4877 cpu_manager.go:414] "containerMap: removing container" podUID="92ba4446-ca90-4760-a2a6-99150b82e360" containerName="revoke-keys"
Oct 30 04:26:11.883032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.882987    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ba4446-ca90-4760-a2a6-99150b82e360" containerName="revoke-keys"
Oct 30 04:26:11.883101 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.883091    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ba4446-ca90-4760-a2a6-99150b82e360" containerName="revoke-keys"
Oct 30 04:26:11.886413 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.886395    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:11.899197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.899171    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:26:11.899197 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.899181    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:26:11.905222 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:11.905197    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"]
Oct 30 04:26:12.010019 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.009984    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.010195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.010045    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.010195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.010174    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rlbc\" (UniqueName: \"kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.111484 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.111392    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.111851 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.111488    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rlbc\" (UniqueName: \"kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.111851 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.111574    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.112024 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.111995    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.113816 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.113793    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.128179 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.128155    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rlbc\" (UniqueName: \"kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc\") pod \"maas-api-revoke-sub-3d4156f1-kcpn4\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.199026 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.198996    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:12.498032 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.498001    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"]
Oct 30 04:26:12.499119 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:26:12.499083    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b0571c_60de_4071_bc07_2c428fe92783.slice/crio-ae027ea0c5228836fdb1596736a23611dc55481776ead5d52d8ba2cded4896d2 WatchSource:0}: Error finding container ae027ea0c5228836fdb1596736a23611dc55481776ead5d52d8ba2cded4896d2: Status 404 returned error can't find the container with id ae027ea0c5228836fdb1596736a23611dc55481776ead5d52d8ba2cded4896d2
Oct 30 04:26:12.538879 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:12.538848    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4" event={"ID":"32b0571c-60de-4071-bc07-2c428fe92783","Type":"ContainerStarted","Data":"ae027ea0c5228836fdb1596736a23611dc55481776ead5d52d8ba2cded4896d2"}
Oct 30 04:26:13.546952 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:13.546915    4877 generic.go:356] "Generic (PLEG): container finished" podID="32b0571c-60de-4071-bc07-2c428fe92783" containerID="a56289fd3614b5b4397ad2e34bbd71d6beba361fb176e56648c7e0ca3d278e00" exitCode=0
Oct 30 04:26:13.547408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:13.546991    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4" event={"ID":"32b0571c-60de-4071-bc07-2c428fe92783","Type":"ContainerDied","Data":"a56289fd3614b5b4397ad2e34bbd71d6beba361fb176e56648c7e0ca3d278e00"}
Oct 30 04:26:13.704943 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:13.704892    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f23594-9d0e-4faf-985b-dbfd67fd9ffd" path="/var/lib/kubelet/pods/54f23594-9d0e-4faf-985b-dbfd67fd9ffd/volumes"
Oct 30 04:26:14.700982 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.700957    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:14.838898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.838805    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc\" (UniqueName: \"kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc\") pod \"32b0571c-60de-4071-bc07-2c428fe92783\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") "
Oct 30 04:26:14.839055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.838901    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca\") pod \"32b0571c-60de-4071-bc07-2c428fe92783\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") "
Oct 30 04:26:14.839055 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.838977    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca\") pod \"32b0571c-60de-4071-bc07-2c428fe92783\" (UID: \"32b0571c-60de-4071-bc07-2c428fe92783\") "
Oct 30 04:26:14.839356 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.839323    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca" pod "32b0571c-60de-4071-bc07-2c428fe92783" (UID: "32b0571c-60de-4071-bc07-2c428fe92783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:14.841194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.841083    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc" pod "32b0571c-60de-4071-bc07-2c428fe92783" (UID: "32b0571c-60de-4071-bc07-2c428fe92783"). InnerVolumeSpecName "kube-api-access-7rlbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:14.841194 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.841181    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca" pod "32b0571c-60de-4071-bc07-2c428fe92783" (UID: "32b0571c-60de-4071-bc07-2c428fe92783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:26:14.940203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.940170    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rlbc\" (UniqueName: \"kubernetes.io/projected/32b0571c-60de-4071-bc07-2c428fe92783-kube-api-access-7rlbc\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:14.940203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.940198    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:14.940203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:14.940208    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0571c-60de-4071-bc07-2c428fe92783-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:15.559797 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.559759    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4" event={"ID":"32b0571c-60de-4071-bc07-2c428fe92783","Type":"ContainerDied","Data":"ae027ea0c5228836fdb1596736a23611dc55481776ead5d52d8ba2cded4896d2"}
Oct 30 04:26:15.559797 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.559779    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"
Oct 30 04:26:15.560095 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.559814    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae027ea0c5228836fdb1596736a23611dc55481776ead5d52d8ba2cded4896d2"
Oct 30 04:26:15.862822 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.862795    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:26:15.950486 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950414    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950520    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950599    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950642    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950697    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950779    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950818    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.950868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950859    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.951100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950870    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:15.951100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950901    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy\") pod \"bb2d08db-9637-4088-8d99-40745a164ca7\" (UID: \"bb2d08db-9637-4088-8d99-40745a164ca7\") "
Oct 30 04:26:15.951100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950952    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:15.951100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950961    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:15.951100 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.950973    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:15.951393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.951206    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:15.951393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.951293    4877 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-credential-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:15.951393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.951314    4877 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-data\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:15.951393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.951328    4877 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2d08db-9637-4088-8d99-40745a164ca7-istiod-ca-cert\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:15.951393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.951341    4877 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-socket\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:15.951393 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.951354    4877 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-workload-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:15.952855 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.952822    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:26:15.953033 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.953000    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "kube-api-access-v8l66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:15.953140 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.953075    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:15.953140 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:15.953077    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token" pod "bb2d08db-9637-4088-8d99-40745a164ca7" (UID: "bb2d08db-9637-4088-8d99-40745a164ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:16.025402 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.025356    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"]
Oct 30 04:26:16.028375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.028346    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fzpm9"]
Oct 30 04:26:16.052374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.052338    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8l66\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-kube-api-access-v8l66\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:16.052374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.052371    4877 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2d08db-9637-4088-8d99-40745a164ca7-istio-token\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:16.052374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.052383    4877 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2d08db-9637-4088-8d99-40745a164ca7-istio-envoy\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:16.052627 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.052391    4877 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2d08db-9637-4088-8d99-40745a164ca7-istio-podinfo\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:16.228462 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:16.228360    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:26:16.568382 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.568349    4877 generic.go:356] "Generic (PLEG): container finished" podID="bb2d08db-9637-4088-8d99-40745a164ca7" containerID="f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773" exitCode=0
Oct 30 04:26:16.568609 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.568401    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"
Oct 30 04:26:16.568609 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.568464    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" event={"ID":"bb2d08db-9637-4088-8d99-40745a164ca7","Type":"ContainerDied","Data":"f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773"}
Oct 30 04:26:16.568609 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.568533    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2" event={"ID":"bb2d08db-9637-4088-8d99-40745a164ca7","Type":"ContainerDied","Data":"f60169d0e1fb7ba46ba899ffbb5eec32f554530a37b133bf4facda94cd34df68"}
Oct 30 04:26:16.568609 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.568565    4877 scope.go:122] "RemoveContainer" containerID="f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773"
Oct 30 04:26:16.579134 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.579113    4877 scope.go:122] "RemoveContainer" containerID="3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"
Oct 30 04:26:16.588390 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.588370    4877 scope.go:122] "RemoveContainer" containerID="f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773"
Oct 30 04:26:16.588693 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:16.588671    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773\": container with ID starting with f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773 not found: ID does not exist" containerID="f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773"
Oct 30 04:26:16.588743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.588703    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773"} err="failed to get container status \"f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773\": rpc error: code = NotFound desc = could not find container \"f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773\": container with ID starting with f94e02d06c55469ee86ffc01bb2a16811d98c30f2b35403651477171829fa773 not found: ID does not exist"
Oct 30 04:26:16.588743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.588722    4877 scope.go:122] "RemoveContainer" containerID="3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"
Oct 30 04:26:16.588998 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:16.588979    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8\": container with ID starting with 3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8 not found: ID does not exist" containerID="3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"
Oct 30 04:26:16.589059 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.589004    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8"} err="failed to get container status \"3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8\": rpc error: code = NotFound desc = could not find container \"3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8\": container with ID starting with 3c5f85bdf0d8cd1f4ee3a32d4b5e981b4c0511686fcc8b7e71459614c9a7a4b8 not found: ID does not exist"
Oct 30 04:26:16.624325 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.624297    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"]
Oct 30 04:26:16.633834 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.633808    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a9dee1-openshift-default-6d74455fb9qdtk2"]
Oct 30 04:26:16.809018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:16.808978    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4" podUID="bc364d2e-0ef0-40e6-b645-d93151d31179" containerName="main" containerID="cri-o://56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702" gracePeriod=45
Oct 30 04:26:17.062882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.062855    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:26:17.163821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.163736    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs\") pod \"bc364d2e-0ef0-40e6-b645-d93151d31179\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") "
Oct 30 04:26:17.163821 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.163778    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf\" (UniqueName: \"kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf\") pod \"bc364d2e-0ef0-40e6-b645-d93151d31179\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") "
Oct 30 04:26:17.164074 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.163835    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache\") pod \"bc364d2e-0ef0-40e6-b645-d93151d31179\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") "
Oct 30 04:26:17.164074 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.163856    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home\") pod \"bc364d2e-0ef0-40e6-b645-d93151d31179\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") "
Oct 30 04:26:17.164074 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.163899    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir\") pod \"bc364d2e-0ef0-40e6-b645-d93151d31179\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") "
Oct 30 04:26:17.164074 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.163940    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm\") pod \"bc364d2e-0ef0-40e6-b645-d93151d31179\" (UID: \"bc364d2e-0ef0-40e6-b645-d93151d31179\") "
Oct 30 04:26:17.164318 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.164087    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache" pod "bc364d2e-0ef0-40e6-b645-d93151d31179" (UID: "bc364d2e-0ef0-40e6-b645-d93151d31179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:17.164318 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.164164    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home" pod "bc364d2e-0ef0-40e6-b645-d93151d31179" (UID: "bc364d2e-0ef0-40e6-b645-d93151d31179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:17.164318 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.164226    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir" pod "bc364d2e-0ef0-40e6-b645-d93151d31179" (UID: "bc364d2e-0ef0-40e6-b645-d93151d31179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:17.164500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.164411    4877 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-tmp-dir\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:17.164500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.164447    4877 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-model-cache\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:17.164500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.164460    4877 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-home\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:17.166174 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.166149    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs" pod "bc364d2e-0ef0-40e6-b645-d93151d31179" (UID: "bc364d2e-0ef0-40e6-b645-d93151d31179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:26:17.166479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.166457    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm" pod "bc364d2e-0ef0-40e6-b645-d93151d31179" (UID: "bc364d2e-0ef0-40e6-b645-d93151d31179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:17.166479 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.166471    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf" pod "bc364d2e-0ef0-40e6-b645-d93151d31179" (UID: "bc364d2e-0ef0-40e6-b645-d93151d31179"). InnerVolumeSpecName "kube-api-access-htcxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:17.265527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.265494    4877 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc364d2e-0ef0-40e6-b645-d93151d31179-tls-certs\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:17.265527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.265522    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htcxf\" (UniqueName: \"kubernetes.io/projected/bc364d2e-0ef0-40e6-b645-d93151d31179-kube-api-access-htcxf\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:17.265527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.265533    4877 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc364d2e-0ef0-40e6-b645-d93151d31179-dshm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:17.576594 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.576563    4877 generic.go:356] "Generic (PLEG): container finished" podID="bc364d2e-0ef0-40e6-b645-d93151d31179" containerID="56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702" exitCode=0
Oct 30 04:26:17.576780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.576609    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"
Oct 30 04:26:17.576780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.576646    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4" event={"ID":"bc364d2e-0ef0-40e6-b645-d93151d31179","Type":"ContainerDied","Data":"56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702"}
Oct 30 04:26:17.576780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.576692    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4" event={"ID":"bc364d2e-0ef0-40e6-b645-d93151d31179","Type":"ContainerDied","Data":"f15ee71b7aaa90fa8da197dd927502b2cc89da163c0d127527606cb35612d5a7"}
Oct 30 04:26:17.576780 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.576715    4877 scope.go:122] "RemoveContainer" containerID="56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702"
Oct 30 04:26:17.586943 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.586923    4877 scope.go:122] "RemoveContainer" containerID="56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702"
Oct 30 04:26:17.587230 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:17.587207    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702\": container with ID starting with 56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702 not found: ID does not exist" containerID="56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702"
Oct 30 04:26:17.587291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.587241    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702"} err="failed to get container status \"56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702\": rpc error: code = NotFound desc = could not find container \"56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702\": container with ID starting with 56563dcf56bb76a09ce529e987082e8c00a6c1123c64422123b1ff00230c7702 not found: ID does not exist"
Oct 30 04:26:17.620366 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.620329    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"]
Oct 30 04:26:17.629905 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.629881    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a9dee1/simulator-single-main-a9dee1-kserve-6557459fc-v6dt4"]
Oct 30 04:26:17.697098 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.697067    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40012a48-1108-4b10-a9e2-f58e225bad91" path="/var/lib/kubelet/pods/40012a48-1108-4b10-a9e2-f58e225bad91/volumes"
Oct 30 04:26:17.697527 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.697512    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" path="/var/lib/kubelet/pods/bb2d08db-9637-4088-8d99-40745a164ca7/volumes"
Oct 30 04:26:17.697893 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:17.697881    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc364d2e-0ef0-40e6-b645-d93151d31179" path="/var/lib/kubelet/pods/bc364d2e-0ef0-40e6-b645-d93151d31179/volumes"
Oct 30 04:26:20.010259 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:20.010215    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:26:21.016619 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:21.016579    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 04:26:28.026259 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:28.026215    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"]
Oct 30 04:26:28.035824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:28.035795    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vfbhw"]
Oct 30 04:26:29.103544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.103519    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:26:29.130245 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.130225    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:26:29.200809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.200773    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume\") pod \"749c0169-3d79-4d0e-8a25-51ef53737676\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") "
Oct 30 04:26:29.201211 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.201187    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume" pod "749c0169-3d79-4d0e-8a25-51ef53737676" (UID: "749c0169-3d79-4d0e-8a25-51ef53737676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:29.301839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.301798    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume\") pod \"a760e24a-c838-41f7-a3fd-92bed4f0108d\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") "
Oct 30 04:26:29.302052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.301853    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln\" (UniqueName: \"kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln\") pod \"749c0169-3d79-4d0e-8a25-51ef53737676\" (UID: \"749c0169-3d79-4d0e-8a25-51ef53737676\") "
Oct 30 04:26:29.302052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.301875    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz\" (UniqueName: \"kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz\") pod \"a760e24a-c838-41f7-a3fd-92bed4f0108d\" (UID: \"a760e24a-c838-41f7-a3fd-92bed4f0108d\") "
Oct 30 04:26:29.302192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.302128    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c0169-3d79-4d0e-8a25-51ef53737676-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.302192 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.302165    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume" pod "a760e24a-c838-41f7-a3fd-92bed4f0108d" (UID: "a760e24a-c838-41f7-a3fd-92bed4f0108d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:26:29.304108 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.304068    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln" pod "749c0169-3d79-4d0e-8a25-51ef53737676" (UID: "749c0169-3d79-4d0e-8a25-51ef53737676"). InnerVolumeSpecName "kube-api-access-rvnln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.304108 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.304089    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz" pod "a760e24a-c838-41f7-a3fd-92bed4f0108d" (UID: "a760e24a-c838-41f7-a3fd-92bed4f0108d"). InnerVolumeSpecName "kube-api-access-mbsfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:29.403347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.403261    4877 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a760e24a-c838-41f7-a3fd-92bed4f0108d-plugins-config-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.403347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.403294    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvnln\" (UniqueName: \"kubernetes.io/projected/749c0169-3d79-4d0e-8a25-51ef53737676-kube-api-access-rvnln\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.403347 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.403307    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsfz\" (UniqueName: \"kubernetes.io/projected/a760e24a-c838-41f7-a3fd-92bed4f0108d-kube-api-access-mbsfz\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:29.669638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.669550    4877 generic.go:356] "Generic (PLEG): container finished" podID="749c0169-3d79-4d0e-8a25-51ef53737676" containerID="250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54" exitCode=0
Oct 30 04:26:29.669638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.669600    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"
Oct 30 04:26:29.669886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.669631    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" event={"ID":"749c0169-3d79-4d0e-8a25-51ef53737676","Type":"ContainerDied","Data":"250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54"}
Oct 30 04:26:29.669886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.669683    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h" event={"ID":"749c0169-3d79-4d0e-8a25-51ef53737676","Type":"ContainerDied","Data":"b197165c9f274b89404e592b4afd59de0858ba7e4996eca3fd8fb7ec45ab97b4"}
Oct 30 04:26:29.669886 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.669705    4877 scope.go:122] "RemoveContainer" containerID="250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54"
Oct 30 04:26:29.671287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.671258    4877 generic.go:356] "Generic (PLEG): container finished" podID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerID="f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8" exitCode=0
Oct 30 04:26:29.671415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.671298    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"
Oct 30 04:26:29.671415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.671357    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" event={"ID":"a760e24a-c838-41f7-a3fd-92bed4f0108d","Type":"ContainerDied","Data":"f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8"}
Oct 30 04:26:29.671415 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.671409    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr" event={"ID":"a760e24a-c838-41f7-a3fd-92bed4f0108d","Type":"ContainerDied","Data":"a230f603b85c0622725f1dbad346888393c43394ac42c997bc79f026c27edc7f"}
Oct 30 04:26:29.681649 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.681632    4877 scope.go:122] "RemoveContainer" containerID="250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54"
Oct 30 04:26:29.681952 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:29.681929    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54\": container with ID starting with 250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54 not found: ID does not exist" containerID="250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54"
Oct 30 04:26:29.682047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.681965    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54"} err="failed to get container status \"250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54\": rpc error: code = NotFound desc = could not find container \"250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54\": container with ID starting with 250343c46ea091ee0e0164def42ccd4c01dd66211b49682059cdeefb18c65f54 not found: ID does not exist"
Oct 30 04:26:29.682047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.681988    4877 scope.go:122] "RemoveContainer" containerID="f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8"
Oct 30 04:26:29.683667 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.683599    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"]
Oct 30 04:26:29.683898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.683834    4877 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6" podUID="adde55dc-547d-42d9-9cc6-7e281e545561" containerName="manager" containerID="cri-o://977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a" gracePeriod=10
Oct 30 04:26:29.693436 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.693403    4877 scope.go:122] "RemoveContainer" containerID="f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8"
Oct 30 04:26:29.693760 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:29.693736    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8\": container with ID starting with f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8 not found: ID does not exist" containerID="f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8"
Oct 30 04:26:29.693840 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.693771    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8"} err="failed to get container status \"f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8\": rpc error: code = NotFound desc = could not find container \"f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8\": container with ID starting with f71b9637dc0e7a87da78b6eb68cd29bfa09f3a56d9e55bcc9f4b30bf34b6c4c8 not found: ID does not exist"
Oct 30 04:26:29.699195 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.699168    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ebf4d8-ade5-4033-8d21-b6ea8101e3eb" path="/var/lib/kubelet/pods/29ebf4d8-ade5-4033-8d21-b6ea8101e3eb/volumes"
Oct 30 04:26:29.716662 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.716638    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"]
Oct 30 04:26:29.720340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.720317    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a9dee1-864f45646jxpxr"]
Oct 30 04:26:29.786814 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.786785    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"]
Oct 30 04:26:29.792277 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:29.792242    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a9dee1-9694948z7h"]
Oct 30 04:26:30.047575 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.047547    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 04:26:30.108638 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.108606    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume\") pod \"adde55dc-547d-42d9-9cc6-7e281e545561\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") "
Oct 30 04:26:30.109172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.108950    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume" pod "adde55dc-547d-42d9-9cc6-7e281e545561" (UID: "adde55dc-547d-42d9-9cc6-7e281e545561"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:26:30.209972 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.209894    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv\" (UniqueName: \"kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv\") pod \"adde55dc-547d-42d9-9cc6-7e281e545561\" (UID: \"adde55dc-547d-42d9-9cc6-7e281e545561\") "
Oct 30 04:26:30.210122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.210073    4877 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adde55dc-547d-42d9-9cc6-7e281e545561-extensions-socket-volume\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:30.211930 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.211905    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv" pod "adde55dc-547d-42d9-9cc6-7e281e545561" (UID: "adde55dc-547d-42d9-9cc6-7e281e545561"). InnerVolumeSpecName "kube-api-access-9jxsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:26:30.311254 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.311221    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxsv\" (UniqueName: \"kubernetes.io/projected/adde55dc-547d-42d9-9cc6-7e281e545561-kube-api-access-9jxsv\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:26:30.681723 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.681687    4877 generic.go:356] "Generic (PLEG): container finished" podID="adde55dc-547d-42d9-9cc6-7e281e545561" containerID="977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a" exitCode=0
Oct 30 04:26:30.681921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.681730    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"
Oct 30 04:26:30.681921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.681741    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6" event={"ID":"adde55dc-547d-42d9-9cc6-7e281e545561","Type":"ContainerDied","Data":"977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a"}
Oct 30 04:26:30.681921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.681785    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6" event={"ID":"adde55dc-547d-42d9-9cc6-7e281e545561","Type":"ContainerDied","Data":"c3a5ad836c8d1446a7ac3778ca9d7c1d8f109696a45381e05d4a8951ffde5093"}
Oct 30 04:26:30.681921 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.681809    4877 scope.go:122] "RemoveContainer" containerID="977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a"
Oct 30 04:26:30.698861 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.698837    4877 scope.go:122] "RemoveContainer" containerID="977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a"
Oct 30 04:26:30.699157 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:30.699134    4877 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a\": container with ID starting with 977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a not found: ID does not exist" containerID="977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a"
Oct 30 04:26:30.699203 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.699166    4877 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a"} err="failed to get container status \"977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a\": rpc error: code = NotFound desc = could not find container \"977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a\": container with ID starting with 977e2c3d54fc7fdd4c14dea68f521df46324cf254743828cc313d90ff66e104a not found: ID does not exist"
Oct 30 04:26:30.736183 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.736153    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"]
Oct 30 04:26:30.746093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:30.746064    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-txkl6"]
Oct 30 04:26:31.696291 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:31.696258    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" path="/var/lib/kubelet/pods/749c0169-3d79-4d0e-8a25-51ef53737676/volumes"
Oct 30 04:26:31.696708 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:31.696656    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" path="/var/lib/kubelet/pods/a760e24a-c838-41f7-a3fd-92bed4f0108d/volumes"
Oct 30 04:26:31.697000 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:31.696986    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adde55dc-547d-42d9-9cc6-7e281e545561" path="/var/lib/kubelet/pods/adde55dc-547d-42d9-9cc6-7e281e545561/volumes"
Oct 30 04:26:46.228589 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:26:46.228544    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:26:56.028640 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:56.028597    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"]
Oct 30 04:26:56.038046 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:56.038017    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-t4qj6"]
Oct 30 04:26:57.697298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:57.697263    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ec346d-05bb-4c33-a7ad-0057297e8727" path="/var/lib/kubelet/pods/a5ec346d-05bb-4c33-a7ad-0057297e8727/volumes"
Oct 30 04:26:58.321956 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:58.321923    4877 scope.go:122] "RemoveContainer" containerID="6a1195e2dba9bf3668f4faf81d89e0fc355a50a7f23815d857dafafe37439440"
Oct 30 04:26:58.331484 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:26:58.331462    4877 scope.go:122] "RemoveContainer" containerID="ed8636842886d9d1643f559195585e80f558887528e78e7aefe3060c675a14ef"
Oct 30 04:27:06.025384 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:06.025345    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"]
Oct 30 04:27:06.027882 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:06.027857    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8t4s7"]
Oct 30 04:27:07.696945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:07.696908    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f7ee7f-688c-4560-8621-be77e605fe4d" path="/var/lib/kubelet/pods/69f7ee7f-688c-4560-8621-be77e605fe4d/volumes"
Oct 30 04:27:16.228505 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:27:16.228465    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:27:28.026136 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:28.026088    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"]
Oct 30 04:27:28.035763 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:28.035733    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-55gpv"]
Oct 30 04:27:29.696853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:29.696819    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c" path="/var/lib/kubelet/pods/62cbbb03-08c9-4fb7-b117-e0cc92eb3f5c/volumes"
Oct 30 04:27:32.934244 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934207    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"]
Oct 30 04:27:32.934845 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934825    4877 cpu_manager.go:414] "containerMap: removing container" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" containerName="payload-pre-processing"
Oct 30 04:27:32.934898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934848    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" containerName="payload-pre-processing"
Oct 30 04:27:32.934898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934867    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy"
Oct 30 04:27:32.934898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934876    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy"
Oct 30 04:27:32.934898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934886    4877 cpu_manager.go:414] "containerMap: removing container" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerName="payload-processing"
Oct 30 04:27:32.934898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934893    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerName="payload-processing"
Oct 30 04:27:32.935063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934922    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bc364d2e-0ef0-40e6-b645-d93151d31179" containerName="main"
Oct 30 04:27:32.935063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934930    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc364d2e-0ef0-40e6-b645-d93151d31179" containerName="main"
Oct 30 04:27:32.935063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934954    4877 cpu_manager.go:414] "containerMap: removing container" podUID="adde55dc-547d-42d9-9cc6-7e281e545561" containerName="manager"
Oct 30 04:27:32.935063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934962    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adde55dc-547d-42d9-9cc6-7e281e545561" containerName="manager"
Oct 30 04:27:32.935063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934977    4877 cpu_manager.go:414] "containerMap: removing container" podUID="32b0571c-60de-4071-bc07-2c428fe92783" containerName="revoke-keys"
Oct 30 04:27:32.935063 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.934985    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b0571c-60de-4071-bc07-2c428fe92783" containerName="revoke-keys"
Oct 30 04:27:32.935252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.935086    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy"
Oct 30 04:27:32.935252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.935101    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc364d2e-0ef0-40e6-b645-d93151d31179" containerName="main"
Oct 30 04:27:32.935252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.935113    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="adde55dc-547d-42d9-9cc6-7e281e545561" containerName="manager"
Oct 30 04:27:32.935252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.935123    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b0571c-60de-4071-bc07-2c428fe92783" containerName="revoke-keys"
Oct 30 04:27:32.935252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.935137    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="749c0169-3d79-4d0e-8a25-51ef53737676" containerName="payload-pre-processing"
Oct 30 04:27:32.935252 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.935147    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="a760e24a-c838-41f7-a3fd-92bed4f0108d" containerName="payload-processing"
Oct 30 04:27:32.939258 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.939232    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:32.967792 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.967759    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cqs85\""
Oct 30 04:27:32.980511 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:32.980484    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"]
Oct 30 04:27:33.015256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.015226    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179cc2a6-a3a9-4f33-a524-95bfabdf7f2d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8blbd\" (UID: \"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.015401 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.015355    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92pk\" (UniqueName: \"kubernetes.io/projected/179cc2a6-a3a9-4f33-a524-95bfabdf7f2d-kube-api-access-z92pk\") pod \"kuadrant-operator-controller-manager-678d87766f-8blbd\" (UID: \"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.116116 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.116076    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92pk\" (UniqueName: \"kubernetes.io/projected/179cc2a6-a3a9-4f33-a524-95bfabdf7f2d-kube-api-access-z92pk\") pod \"kuadrant-operator-controller-manager-678d87766f-8blbd\" (UID: \"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.116310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.116147    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179cc2a6-a3a9-4f33-a524-95bfabdf7f2d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8blbd\" (UID: \"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.116653 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.116632    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179cc2a6-a3a9-4f33-a524-95bfabdf7f2d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8blbd\" (UID: \"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.137710 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.137676    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92pk\" (UniqueName: \"kubernetes.io/projected/179cc2a6-a3a9-4f33-a524-95bfabdf7f2d-kube-api-access-z92pk\") pod \"kuadrant-operator-controller-manager-678d87766f-8blbd\" (UID: \"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.252189 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.252113    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:33.589824 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:33.589792    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"]
Oct 30 04:27:33.590408 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:27:33.590383    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179cc2a6_a3a9_4f33_a524_95bfabdf7f2d.slice/crio-5fa69bce66a90ee81115609f0a50daf2d37cac9e60219bc77a1a97ae9bc2e5f4 WatchSource:0}: Error finding container 5fa69bce66a90ee81115609f0a50daf2d37cac9e60219bc77a1a97ae9bc2e5f4: Status 404 returned error can't find the container with id 5fa69bce66a90ee81115609f0a50daf2d37cac9e60219bc77a1a97ae9bc2e5f4
Oct 30 04:27:34.081875 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:34.081839    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd" event={"ID":"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d","Type":"ContainerStarted","Data":"60e1e2fb741855888d5cdff19311510f19ad4042b1178ad42625d77f067d6d47"}
Oct 30 04:27:34.082345 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:34.081894    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd" event={"ID":"179cc2a6-a3a9-4f33-a524-95bfabdf7f2d","Type":"ContainerStarted","Data":"5fa69bce66a90ee81115609f0a50daf2d37cac9e60219bc77a1a97ae9bc2e5f4"}
Oct 30 04:27:34.082345 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:34.081930    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:34.121659 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:34.121608    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd" podStartSLOduration=2.121595653 podStartE2EDuration="2.121595653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:34.117834916 +0000 UTC m=+5082.546724355" watchObservedRunningTime="2026-10-30 04:27:34.121595653 +0000 UTC m=+5082.550485070"
Oct 30 04:27:45.024228 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:45.024195    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"]
Oct 30 04:27:45.033596 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:45.033569    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z8zzp"]
Oct 30 04:27:45.090950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:45.090922    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8blbd"
Oct 30 04:27:45.696744 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:45.696709    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0878fcf1-a7de-44e5-a297-38fee37dac68" path="/var/lib/kubelet/pods/0878fcf1-a7de-44e5-a297-38fee37dac68/volumes"
Oct 30 04:27:46.227917 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:27:46.227879    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:27:53.235570 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:53.235532    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:27:53.263668 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:53.263619    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:27:56.873978 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.873941    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"]
Oct 30 04:27:56.874478 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.874462    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy"
Oct 30 04:27:56.874569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.874480    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy"
Oct 30 04:27:56.874569 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.874558    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2d08db-9637-4088-8d99-40745a164ca7" containerName="istio-proxy"
Oct 30 04:27:56.877936 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.877906    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.883986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.883956    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.884122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.884023    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.884122 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.884100    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbr8\" (UniqueName: \"kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.897938 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.897886    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"]
Oct 30 04:27:56.903588 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.903555    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:27:56.904165 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.904143    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:27:56.984887 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.984835    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.985127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.984899    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.985127 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.984934    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbr8\" (UniqueName: \"kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.985728 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.985697    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:56.987389 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:56.987368    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:57.002718 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:57.002693    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbr8\" (UniqueName: \"kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8\") pod \"maas-api-revoke-sub-d576e68b-q2lbg\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:57.191908 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:57.191823    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:27:57.489844 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:57.489814    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"]
Oct 30 04:27:57.491657 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:27:57.491627    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff5b122_8fce_41ae_9ccb_30e12ffb4f41.slice/crio-95520c7d41bd0f23935871b2a90ecdaf59469cf3f10ac4957eb79e04aeeb2a24 WatchSource:0}: Error finding container 95520c7d41bd0f23935871b2a90ecdaf59469cf3f10ac4957eb79e04aeeb2a24: Status 404 returned error can't find the container with id 95520c7d41bd0f23935871b2a90ecdaf59469cf3f10ac4957eb79e04aeeb2a24
Oct 30 04:27:58.278454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:58.278394    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg" event={"ID":"bff5b122-8fce-41ae-9ccb-30e12ffb4f41","Type":"ContainerStarted","Data":"6be18ec3e485cfd1814b25a6bbd00ac07e3d7fb717a13022d1931518afb91861"}
Oct 30 04:27:58.278929 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:58.278474    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg" event={"ID":"bff5b122-8fce-41ae-9ccb-30e12ffb4f41","Type":"ContainerStarted","Data":"95520c7d41bd0f23935871b2a90ecdaf59469cf3f10ac4957eb79e04aeeb2a24"}
Oct 30 04:27:58.314130 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:58.314075    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg" podStartSLOduration=2.314060207 podStartE2EDuration="2.314060207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:58.312172259 +0000 UTC m=+5106.741061697" watchObservedRunningTime="2026-10-30 04:27:58.314060207 +0000 UTC m=+5106.742949616"
Oct 30 04:27:58.385196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:58.385169    4877 scope.go:122] "RemoveContainer" containerID="f80174e65838dfd01eedbb9545351f213ace938da732237edeb4ab6fda63920c"
Oct 30 04:27:58.453239 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:58.453213    4877 scope.go:122] "RemoveContainer" containerID="07084438a4ed706231b6ef828fe13c4e9fc9d51a641510d9ec0dd94c9ba381bf"
Oct 30 04:27:58.461925 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:58.461903    4877 scope.go:122] "RemoveContainer" containerID="cefd832c3e042ee37b82f7f20919bde1ea4cd61b2110da4289b1ef1c1af3977a"
Oct 30 04:27:59.286152 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:59.286111    4877 generic.go:356] "Generic (PLEG): container finished" podID="bff5b122-8fce-41ae-9ccb-30e12ffb4f41" containerID="6be18ec3e485cfd1814b25a6bbd00ac07e3d7fb717a13022d1931518afb91861" exitCode=0
Oct 30 04:27:59.286699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:27:59.286200    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg" event={"ID":"bff5b122-8fce-41ae-9ccb-30e12ffb4f41","Type":"ContainerDied","Data":"6be18ec3e485cfd1814b25a6bbd00ac07e3d7fb717a13022d1931518afb91861"}
Oct 30 04:28:00.448599 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.448572    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:28:00.516989 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.516954    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8\" (UniqueName: \"kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8\") pod \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") "
Oct 30 04:28:00.517221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.517123    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca\") pod \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") "
Oct 30 04:28:00.517221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.517183    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca\") pod \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\" (UID: \"bff5b122-8fce-41ae-9ccb-30e12ffb4f41\") "
Oct 30 04:28:00.517584 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.517551    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca" pod "bff5b122-8fce-41ae-9ccb-30e12ffb4f41" (UID: "bff5b122-8fce-41ae-9ccb-30e12ffb4f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:28:00.519104 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.519076    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8" pod "bff5b122-8fce-41ae-9ccb-30e12ffb4f41" (UID: "bff5b122-8fce-41ae-9ccb-30e12ffb4f41"). InnerVolumeSpecName "kube-api-access-wzbr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:28:00.519196 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.519173    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca" pod "bff5b122-8fce-41ae-9ccb-30e12ffb4f41" (UID: "bff5b122-8fce-41ae-9ccb-30e12ffb4f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:28:00.619011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.618924    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:28:00.619011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.618957    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:28:00.619011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:00.618966    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbr8\" (UniqueName: \"kubernetes.io/projected/bff5b122-8fce-41ae-9ccb-30e12ffb4f41-kube-api-access-wzbr8\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:28:01.299658 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:01.299624    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg" event={"ID":"bff5b122-8fce-41ae-9ccb-30e12ffb4f41","Type":"ContainerDied","Data":"95520c7d41bd0f23935871b2a90ecdaf59469cf3f10ac4957eb79e04aeeb2a24"}
Oct 30 04:28:01.299852 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:01.299661    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"
Oct 30 04:28:01.299852 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:01.299673    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95520c7d41bd0f23935871b2a90ecdaf59469cf3f10ac4957eb79e04aeeb2a24"
Oct 30 04:28:03.025305 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:03.025273    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"]
Oct 30 04:28:03.034804 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:03.034774    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-8982t"]
Oct 30 04:28:03.697288 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:03.697246    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d608f18-adfc-4d76-a360-56df17b3d764" path="/var/lib/kubelet/pods/6d608f18-adfc-4d76-a360-56df17b3d764/volumes"
Oct 30 04:28:14.381950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.381908    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"]
Oct 30 04:28:14.382564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.382546    4877 cpu_manager.go:414] "containerMap: removing container" podUID="bff5b122-8fce-41ae-9ccb-30e12ffb4f41" containerName="revoke-keys"
Oct 30 04:28:14.382564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.382565    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff5b122-8fce-41ae-9ccb-30e12ffb4f41" containerName="revoke-keys"
Oct 30 04:28:14.382727 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.382661    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff5b122-8fce-41ae-9ccb-30e12ffb4f41" containerName="revoke-keys"
Oct 30 04:28:14.385945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.385923    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.408418 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.408385    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:28:14.408625 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.408600    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:28:14.413000 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.412973    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"]
Oct 30 04:28:14.474500 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.474463    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhfm\" (UniqueName: \"kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.474686 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.474587    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.474686 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.474644    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.575623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.575584    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.575623 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.575623    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.575906 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.575702    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhfm\" (UniqueName: \"kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.576271 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.576247    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.578066 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.578047    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.592743 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.592713    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhfm\" (UniqueName: \"kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm\") pod \"maas-api-revoke-sub-d2bc93bc-wqj59\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:14.706751 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:14.706671    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:15.003372 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:15.003346    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"]
Oct 30 04:28:15.005662 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:28:15.005634    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2565c4_5956_41e4_b1ac_5def850abe8e.slice/crio-9473bab20439874c1084707f754cb27c6580913bbf005d17d5577850a1014af4 WatchSource:0}: Error finding container 9473bab20439874c1084707f754cb27c6580913bbf005d17d5577850a1014af4: Status 404 returned error can't find the container with id 9473bab20439874c1084707f754cb27c6580913bbf005d17d5577850a1014af4
Oct 30 04:28:15.394484 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:15.394452    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59" event={"ID":"aa2565c4-5956-41e4-b1ac-5def850abe8e","Type":"ContainerStarted","Data":"70e80ec17751ff4ddd0c0cefe7bbf37988c1d68c6b85ef869cd6513268b8314b"}
Oct 30 04:28:15.394895 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:15.394513    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59" event={"ID":"aa2565c4-5956-41e4-b1ac-5def850abe8e","Type":"ContainerStarted","Data":"9473bab20439874c1084707f754cb27c6580913bbf005d17d5577850a1014af4"}
Oct 30 04:28:15.429359 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:15.429296    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59" podStartSLOduration=1.429282747 podStartE2EDuration="1.429282747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:15.427583138 +0000 UTC m=+5123.856472559" watchObservedRunningTime="2026-10-30 04:28:15.429282747 +0000 UTC m=+5123.858172163"
Oct 30 04:28:16.228502 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:28:16.228407    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:28:16.401999 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:16.401965    4877 generic.go:356] "Generic (PLEG): container finished" podID="aa2565c4-5956-41e4-b1ac-5def850abe8e" containerID="70e80ec17751ff4ddd0c0cefe7bbf37988c1d68c6b85ef869cd6513268b8314b" exitCode=0
Oct 30 04:28:16.402408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:16.402050    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59" event={"ID":"aa2565c4-5956-41e4-b1ac-5def850abe8e","Type":"ContainerDied","Data":"70e80ec17751ff4ddd0c0cefe7bbf37988c1d68c6b85ef869cd6513268b8314b"}
Oct 30 04:28:17.024770 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.024734    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"]
Oct 30 04:28:17.034256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.034223    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jb2f6"]
Oct 30 04:28:17.556110 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.556085    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:17.610986 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.610957    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm\" (UniqueName: \"kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm\") pod \"aa2565c4-5956-41e4-b1ac-5def850abe8e\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") "
Oct 30 04:28:17.611161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.611056    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca\") pod \"aa2565c4-5956-41e4-b1ac-5def850abe8e\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") "
Oct 30 04:28:17.611217 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.611187    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca\") pod \"aa2565c4-5956-41e4-b1ac-5def850abe8e\" (UID: \"aa2565c4-5956-41e4-b1ac-5def850abe8e\") "
Oct 30 04:28:17.611537 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.611492    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca" pod "aa2565c4-5956-41e4-b1ac-5def850abe8e" (UID: "aa2565c4-5956-41e4-b1ac-5def850abe8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:28:17.613279 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.613250    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca" pod "aa2565c4-5956-41e4-b1ac-5def850abe8e" (UID: "aa2565c4-5956-41e4-b1ac-5def850abe8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:28:17.613371 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.613319    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm" pod "aa2565c4-5956-41e4-b1ac-5def850abe8e" (UID: "aa2565c4-5956-41e4-b1ac-5def850abe8e"). InnerVolumeSpecName "kube-api-access-gzhfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:28:17.699089 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.699003    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45176c6e-2969-4751-ad02-5a0775d12981" path="/var/lib/kubelet/pods/45176c6e-2969-4751-ad02-5a0775d12981/volumes"
Oct 30 04:28:17.712139 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.712101    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:28:17.712139 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.712134    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzhfm\" (UniqueName: \"kubernetes.io/projected/aa2565c4-5956-41e4-b1ac-5def850abe8e-kube-api-access-gzhfm\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:28:17.712350 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:17.712150    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2565c4-5956-41e4-b1ac-5def850abe8e-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:28:18.416497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:18.416464    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"
Oct 30 04:28:18.416497 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:18.416484    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59" event={"ID":"aa2565c4-5956-41e4-b1ac-5def850abe8e","Type":"ContainerDied","Data":"9473bab20439874c1084707f754cb27c6580913bbf005d17d5577850a1014af4"}
Oct 30 04:28:18.416729 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:18.416548    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9473bab20439874c1084707f754cb27c6580913bbf005d17d5577850a1014af4"
Oct 30 04:28:34.751295 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.751256    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"]
Oct 30 04:28:34.751988 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.751966    4877 cpu_manager.go:414] "containerMap: removing container" podUID="aa2565c4-5956-41e4-b1ac-5def850abe8e" containerName="revoke-keys"
Oct 30 04:28:34.752043 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.751992    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2565c4-5956-41e4-b1ac-5def850abe8e" containerName="revoke-keys"
Oct 30 04:28:34.752130 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.752119    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2565c4-5956-41e4-b1ac-5def850abe8e" containerName="revoke-keys"
Oct 30 04:28:34.757093 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.757073    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.773031 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.773005    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"]
Oct 30 04:28:34.779642 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.779613    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 04:28:34.782380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.782356    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 04:28:34.782380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.782357    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8gxpp\""
Oct 30 04:28:34.878871 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.878841    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-openshift-service-ca\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.879052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.878880    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-maas-api-tls\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.879052 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.878916    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt574\" (UniqueName: \"kubernetes.io/projected/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-kube-api-access-lt574\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.879135 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.879048    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-metrics-certs\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.980062 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.980015    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt574\" (UniqueName: \"kubernetes.io/projected/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-kube-api-access-lt574\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.980255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.980129    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-metrics-certs\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.980255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.980153    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-openshift-service-ca\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.980255 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.980180    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-maas-api-tls\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.981021 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.980999    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-openshift-service-ca\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.982738 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.982712    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-maas-api-tls\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.982851 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.982736    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-metrics-certs\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:34.997283 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:34.997252    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt574\" (UniqueName: \"kubernetes.io/projected/42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0-kube-api-access-lt574\") pod \"maas-api-bcb58fd6d-rw484\" (UID: \"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:35.071011 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:35.070980    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:35.370633 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:35.370601    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"]
Oct 30 04:28:35.371581 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:28:35.371554    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b3c94d_5ca0_45a6_9ae9_d2f63c0e19e0.slice/crio-ccfb5a9513781e3915fec7386782290a341be02cd64141f18077cd6e31d79c40 WatchSource:0}: Error finding container ccfb5a9513781e3915fec7386782290a341be02cd64141f18077cd6e31d79c40: Status 404 returned error can't find the container with id ccfb5a9513781e3915fec7386782290a341be02cd64141f18077cd6e31d79c40
Oct 30 04:28:35.372904 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:35.372883    4877 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:28:35.533801 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:35.533762    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484" event={"ID":"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0","Type":"ContainerStarted","Data":"ccfb5a9513781e3915fec7386782290a341be02cd64141f18077cd6e31d79c40"}
Oct 30 04:28:38.561322 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:38.561275    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484" event={"ID":"42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0","Type":"ContainerStarted","Data":"634e4770db9f658a1fd1b77691a0f7f12a1b4b47c8861548692b0d5a69f9377c"}
Oct 30 04:28:38.561843 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:38.561439    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:38.601520 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:38.601455    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484" podStartSLOduration=2.332040336 podStartE2EDuration="4.601437263s" totalImagesPullingTime="2.269396927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:38.598007325 +0000 UTC m=+5147.026896759" watchObservedRunningTime="2026-10-30 04:28:38.601437263 +0000 UTC m=+5147.030326674"
Oct 30 04:28:44.578254 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:44.578223    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-rw484"
Oct 30 04:28:46.228485 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:28:46.228445    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:28:58.493182 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:58.493129    4877 scope.go:122] "RemoveContainer" containerID="b210a18b4085f91c99c4a00560217f571c275805ed949b4ef77c6f69ee5e2853"
Oct 30 04:28:58.502517 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:58.502492    4877 scope.go:122] "RemoveContainer" containerID="154dcc2781fa40ab442662f9374eced0cc3bdd96dde8774017cbedf09d94fa0b"
Oct 30 04:28:58.510894 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:28:58.510873    4877 scope.go:122] "RemoveContainer" containerID="ed7b309ef0eb4c7de5a659cdb6bff65c606625d0dc146716ed40ee345862a5fa"
Oct 30 04:29:14.051610 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:14.051513    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"]
Oct 30 04:29:14.054946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:14.054916    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-g644n"]
Oct 30 04:29:15.697380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:15.697342    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eea2de9-d4ff-41de-a861-afcaef18c9bf" path="/var/lib/kubelet/pods/0eea2de9-d4ff-41de-a861-afcaef18c9bf/volumes"
Oct 30 04:29:16.229969 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:29:16.229928    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:29:25.026120 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:25.026088    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"]
Oct 30 04:29:25.035351 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:25.035327    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t74s8"]
Oct 30 04:29:25.697047 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:25.697011    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9709babe-5874-4d63-9bdb-dd7548127c7f" path="/var/lib/kubelet/pods/9709babe-5874-4d63-9bdb-dd7548127c7f/volumes"
Oct 30 04:29:46.245561 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:29:46.245524    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:29:58.541013 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:58.540980    4877 scope.go:122] "RemoveContainer" containerID="480f20e4b61a5f06cbafc82f12288a4bc2c0973f1c3095f18ab0f42475485e1f"
Oct 30 04:29:58.550501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:58.550483    4877 scope.go:122] "RemoveContainer" containerID="99bf71ae10a3d3e1681d1a9c29399df1cc0d15f77709adef4e606847688de805"
Oct 30 04:29:58.559302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:58.559286    4877 scope.go:122] "RemoveContainer" containerID="59127ccb0ff3bc42888ce9ee0fbe9cd25fdb0611f62992a909e48658d185f874"
Oct 30 04:29:58.566901 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:29:58.566883    4877 scope.go:122] "RemoveContainer" containerID="974ddcd49cf89b8d17ecebb705a342f5249797da5e074edb5a14229ff0cfaa13"
Oct 30 04:30:00.481221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.481185    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"]
Oct 30 04:30:00.486375 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.486357    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.500368 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.500318    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 04:30:00.503731 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.503698    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"]
Oct 30 04:30:00.510209 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.510183    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-grvs9\""
Oct 30 04:30:00.575249 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.575212    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.575472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.575271    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6659d\" (UniqueName: \"kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.575472 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.575451    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.676348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.676304    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.676348 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.676368    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6659d\" (UniqueName: \"kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.676719 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.676481    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.677121 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.677094    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.678839 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.678808    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.693915 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.693889    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6659d\" (UniqueName: \"kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d\") pod \"maas-api-revoke-sub-374aabff-6k7b4\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:00.799380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:00.799347    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:01.101573 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:01.101542    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"]
Oct 30 04:30:01.103836 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:30:01.103809    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c9bd0b_2448_4fc0_8fb3_60c622ed5034.slice/crio-8a06edbf3f3956d966013d1dd0d05c40ddbe98868d7f26167539dc331a998809 WatchSource:0}: Error finding container 8a06edbf3f3956d966013d1dd0d05c40ddbe98868d7f26167539dc331a998809: Status 404 returned error can't find the container with id 8a06edbf3f3956d966013d1dd0d05c40ddbe98868d7f26167539dc331a998809
Oct 30 04:30:01.189148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:01.189117    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4" event={"ID":"80c9bd0b-2448-4fc0-8fb3-60c622ed5034","Type":"ContainerStarted","Data":"8a06edbf3f3956d966013d1dd0d05c40ddbe98868d7f26167539dc331a998809"}
Oct 30 04:30:02.197256 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:02.197218    4877 generic.go:356] "Generic (PLEG): container finished" podID="80c9bd0b-2448-4fc0-8fb3-60c622ed5034" containerID="15a57c7b6b87572c4c623ee25dadbb5c4a69d1b749e550c4c2bbe5757c53bf9e" exitCode=0
Oct 30 04:30:02.197715 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:02.197264    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4" event={"ID":"80c9bd0b-2448-4fc0-8fb3-60c622ed5034","Type":"ContainerDied","Data":"15a57c7b6b87572c4c623ee25dadbb5c4a69d1b749e550c4c2bbe5757c53bf9e"}
Oct 30 04:30:03.337090 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.337065    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:03.512771 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.512683    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca\") pod \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") "
Oct 30 04:30:03.512974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.512777    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d\" (UniqueName: \"kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d\") pod \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") "
Oct 30 04:30:03.512974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.512822    4877 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca\") pod \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\" (UID: \"80c9bd0b-2448-4fc0-8fb3-60c622ed5034\") "
Oct 30 04:30:03.513325 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.513294    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca" pod "80c9bd0b-2448-4fc0-8fb3-60c622ed5034" (UID: "80c9bd0b-2448-4fc0-8fb3-60c622ed5034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:30:03.515721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.515694    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca" pod "80c9bd0b-2448-4fc0-8fb3-60c622ed5034" (UID: "80c9bd0b-2448-4fc0-8fb3-60c622ed5034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:30:03.515721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.515714    4877 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d" pod "80c9bd0b-2448-4fc0-8fb3-60c622ed5034" (UID: "80c9bd0b-2448-4fc0-8fb3-60c622ed5034"). InnerVolumeSpecName "kube-api-access-6659d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:30:03.614583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.614542    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-xks-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:30:03.614583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.614576    4877 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6659d\" (UniqueName: \"kubernetes.io/projected/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-kube-api-access-6659d\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:30:03.614583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:03.614592    4877 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c9bd0b-2448-4fc0-8fb3-60c622ed5034-maas-api-openshift-ca\") on node \"ip-10-0-135-226.ec2.internal\" DevicePath \"\""
Oct 30 04:30:04.211221 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:04.211184    4877 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4"
Oct 30 04:30:04.211434 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:04.211250    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374aabff-6k7b4" event={"ID":"80c9bd0b-2448-4fc0-8fb3-60c622ed5034","Type":"ContainerDied","Data":"8a06edbf3f3956d966013d1dd0d05c40ddbe98868d7f26167539dc331a998809"}
Oct 30 04:30:04.211434 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:04.211304    4877 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a06edbf3f3956d966013d1dd0d05c40ddbe98868d7f26167539dc331a998809"
Oct 30 04:30:16.267415 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:30:16.267374    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:30:46.229020 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:30:46.228920    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:30:49.026172 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:49.026138    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"]
Oct 30 04:30:49.035376 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:49.035347    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef440a36-9rvcj"]
Oct 30 04:30:49.696825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:49.696787    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ba4446-ca90-4760-a2a6-99150b82e360" path="/var/lib/kubelet/pods/92ba4446-ca90-4760-a2a6-99150b82e360/volumes"
Oct 30 04:30:58.600501 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:58.600471    4877 scope.go:122] "RemoveContainer" containerID="08cdad42f6c6e1d71196b4ab8dad9c707fcb2c7e1e2a1f06accb42c5288c3b75"
Oct 30 04:30:58.610782 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:58.610756    4877 scope.go:122] "RemoveContainer" containerID="c41f0a4c0e12b18582612ab262d0f7a5da9b11786663991347e344453e9f1162"
Oct 30 04:30:58.619298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:30:58.619273    4877 scope.go:122] "RemoveContainer" containerID="3bb34604429dff7e814a0ed0fe0f11480d88cd82c498d11c66f7d46de69f2373"
Oct 30 04:31:05.888586 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:31:05.888534    4877 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 04:31:05.889240 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:31:05.889199    4877 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qzl6k" podUID="1057196a-2b55-42a3-a8d4-4918a927d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:31:15.025946 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:31:15.025909    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"]
Oct 30 04:31:15.035541 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:31:15.035507    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kcpn4"]
Oct 30 04:31:15.705287 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:31:15.705247    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b0571c-60de-4071-bc07-2c428fe92783" path="/var/lib/kubelet/pods/32b0571c-60de-4071-bc07-2c428fe92783/volumes"
Oct 30 04:31:16.228756 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:31:16.228717    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:31:46.275605 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:31:46.275564    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:31:58.648953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:31:58.648919    4877 scope.go:122] "RemoveContainer" containerID="77ad9bb214b23211af2445d864ec73fb1e656d49e1908f0bc00942e848a00357"
Oct 30 04:32:16.229016 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:32:16.228975    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:32:46.228687 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:32:46.228645    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:32:53.278374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:32:53.278342    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:32:53.309471 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:32:53.309447    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:32:58.677442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:32:58.677374    4877 scope.go:122] "RemoveContainer" containerID="a56289fd3614b5b4397ad2e34bbd71d6beba361fb176e56648c7e0ca3d278e00"
Oct 30 04:33:01.024162 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:01.024127    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"]
Oct 30 04:33:01.033647 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:01.033620    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-q2lbg"]
Oct 30 04:33:01.698310 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:01.698276    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff5b122-8fce-41ae-9ccb-30e12ffb4f41" path="/var/lib/kubelet/pods/bff5b122-8fce-41ae-9ccb-30e12ffb4f41/volumes"
Oct 30 04:33:16.229206 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:33:16.229166    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:33:18.028091 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:18.028058    4877 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"]
Oct 30 04:33:18.037174 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:18.037143    4877 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-wqj59"]
Oct 30 04:33:19.716458 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:19.713740    4877 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2565c4-5956-41e4-b1ac-5def850abe8e" path="/var/lib/kubelet/pods/aa2565c4-5956-41e4-b1ac-5def850abe8e/volumes"
Oct 30 04:33:27.035579 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:27.035538    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b66c856-5qx55_f34c49c1-2d78-4b79-8505-f68314e4d4b9/authorino/0.log"
Oct 30 04:33:40.718699 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:40.718603    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:33:40.957302 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:40.957252    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:33:41.916128 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:41.916094    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bcb58fd6d-rw484_42b3c94d-5ca0-45a6-9ae9-d2f63c0e19e0/maas-api/0.log"
Oct 30 04:33:42.948508 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:42.948478    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-374aabff-6k7b4_80c9bd0b-2448-4fc0-8fb3-60c622ed5034/revoke-keys/0.log"
Oct 30 04:33:45.985036 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:45.985000    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-9pfkh_a51876bb-942d-483d-9cda-5372b1b0399f/postgres/0.log"
Oct 30 04:33:46.229118 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:33:46.229076    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:33:47.983898 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:47.983849    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b66c856-5qx55_f34c49c1-2d78-4b79-8505-f68314e4d4b9/authorino/0.log"
Oct 30 04:33:48.211798 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:48.211764    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-npmx5_297ec4ff-cbaf-48ea-a5cd-fab91d07a75e/manager/0.log"
Oct 30 04:33:48.411074 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:48.411040    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gggrh_713ca209-955a-44dd-b634-18e58ef6735b/manager/0.log"
Oct 30 04:33:48.595682 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:48.595655    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8m78v_490cd1fd-4982-4de2-824f-1a3f359dc24d/kuadrant-console-plugin/0.log"
Oct 30 04:33:48.837604 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:48.837571    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8blbd_179cc2a6-a3a9-4f33-a524-95bfabdf7f2d/manager/0.log"
Oct 30 04:33:49.293373 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:49.293329    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rcpz6_66695834-c9cd-430e-946c-aa57ae794bca/manager/0.log"
Oct 30 04:33:50.283298 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:50.283268    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn_1a4cb6aa-0810-497b-a9c8-510595534f09/istio-proxy/0.log"
Oct 30 04:33:51.502016 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:51.501966    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/1.log"
Oct 30 04:33:51.722583 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:51.722554    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fs5rj_11ff9767-6324-495f-be0e-6bca50031925/istio-proxy/0.log"
Oct 30 04:33:52.143288 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:52.143251    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-qzl6k_1057196a-2b55-42a3-a8d4-4918a927d4b7/istio-proxy/0.log"
Oct 30 04:33:53.168796 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:53.168762    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm_cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c/istio-proxy/0.log"
Oct 30 04:33:53.731280 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:53.731247    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ddf86b9f-2wfp9_1569b662-064a-4998-b332-03c6ed48e312/router/0.log"
Oct 30 04:33:54.382740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:54.382705    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/0.log"
Oct 30 04:33:55.108177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:55.108143    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vvg6r_c4e4abcc-bfca-4458-8aed-8b4fca74e552/main/0.log"
Oct 30 04:33:55.302301 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:55.302266    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-j59xx_41030117-c589-4881-8d38-fe2d36adbc70/main/0.log"
Oct 30 04:33:55.685134 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:55.685100    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tn8c5_b250ae5d-958a-4a5b-9eee-3936a1f2ec2a/main/0.log"
Oct 30 04:33:56.099784 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:56.099756    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-h5wsn_b8e0b1b4-6014-40fc-a926-bc3780736c52/main/0.log"
Oct 30 04:33:58.835191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:33:58.835159    4877 scope.go:122] "RemoveContainer" containerID="6be18ec3e485cfd1814b25a6bbd00ac07e3d7fb717a13022d1931518afb91861"
Oct 30 04:34:07.382177 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.382141    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k586c/must-gather-5gktq"]
Oct 30 04:34:07.382687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.382663    4877 cpu_manager.go:414] "containerMap: removing container" podUID="80c9bd0b-2448-4fc0-8fb3-60c622ed5034" containerName="revoke-keys"
Oct 30 04:34:07.382687 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.382686    4877 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c9bd0b-2448-4fc0-8fb3-60c622ed5034" containerName="revoke-keys"
Oct 30 04:34:07.382864 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.382788    4877 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c9bd0b-2448-4fc0-8fb3-60c622ed5034" containerName="revoke-keys"
Oct 30 04:34:07.387551 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.387532    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.402204 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.402176    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k586c/must-gather-5gktq"]
Oct 30 04:34:07.405447 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.405408    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k586c\"/\"default-dockercfg-hmwml\""
Oct 30 04:34:07.409635 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.409612    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k586c\"/\"kube-root-ca.crt\""
Oct 30 04:34:07.409747 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.409654    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k586c\"/\"openshift-service-ca.crt\""
Oct 30 04:34:07.455881 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.455837    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgrj\" (UniqueName: \"kubernetes.io/projected/d042d099-b465-4319-9f62-f32c2267edb0-kube-api-access-5vgrj\") pod \"must-gather-5gktq\" (UID: \"d042d099-b465-4319-9f62-f32c2267edb0\") " pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.456041 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.455970    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d042d099-b465-4319-9f62-f32c2267edb0-must-gather-output\") pod \"must-gather-5gktq\" (UID: \"d042d099-b465-4319-9f62-f32c2267edb0\") " pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.477290 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.477259    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9nqc/must-gather-8fkp8"]
Oct 30 04:34:07.487396 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.487363    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.499206 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.499180    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/must-gather-8fkp8"]
Oct 30 04:34:07.505510 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.505482    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nqc\"/\"openshift-service-ca.crt\""
Oct 30 04:34:07.505632 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.505604    4877 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nqc\"/\"kube-root-ca.crt\""
Oct 30 04:34:07.505704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.505665    4877 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s9nqc\"/\"default-dockercfg-kgslt\""
Oct 30 04:34:07.557018 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.556986    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d042d099-b465-4319-9f62-f32c2267edb0-must-gather-output\") pod \"must-gather-5gktq\" (UID: \"d042d099-b465-4319-9f62-f32c2267edb0\") " pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.557161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.557047    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ea2cc76-4433-43d6-ba70-8f753e4a16a8-must-gather-output\") pod \"must-gather-8fkp8\" (UID: \"2ea2cc76-4433-43d6-ba70-8f753e4a16a8\") " pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.557161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.557097    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgrj\" (UniqueName: \"kubernetes.io/projected/d042d099-b465-4319-9f62-f32c2267edb0-kube-api-access-5vgrj\") pod \"must-gather-5gktq\" (UID: \"d042d099-b465-4319-9f62-f32c2267edb0\") " pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.557161 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.557127    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvktj\" (UniqueName: \"kubernetes.io/projected/2ea2cc76-4433-43d6-ba70-8f753e4a16a8-kube-api-access-wvktj\") pod \"must-gather-8fkp8\" (UID: \"2ea2cc76-4433-43d6-ba70-8f753e4a16a8\") " pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.557363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.557341    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d042d099-b465-4319-9f62-f32c2267edb0-must-gather-output\") pod \"must-gather-5gktq\" (UID: \"d042d099-b465-4319-9f62-f32c2267edb0\") " pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.573868 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.573839    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgrj\" (UniqueName: \"kubernetes.io/projected/d042d099-b465-4319-9f62-f32c2267edb0-kube-api-access-5vgrj\") pod \"must-gather-5gktq\" (UID: \"d042d099-b465-4319-9f62-f32c2267edb0\") " pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.658059 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.657996    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ea2cc76-4433-43d6-ba70-8f753e4a16a8-must-gather-output\") pod \"must-gather-8fkp8\" (UID: \"2ea2cc76-4433-43d6-ba70-8f753e4a16a8\") " pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.658191 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.658068    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvktj\" (UniqueName: \"kubernetes.io/projected/2ea2cc76-4433-43d6-ba70-8f753e4a16a8-kube-api-access-wvktj\") pod \"must-gather-8fkp8\" (UID: \"2ea2cc76-4433-43d6-ba70-8f753e4a16a8\") " pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.658371 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.658350    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2ea2cc76-4433-43d6-ba70-8f753e4a16a8-must-gather-output\") pod \"must-gather-8fkp8\" (UID: \"2ea2cc76-4433-43d6-ba70-8f753e4a16a8\") " pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.674740 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.674719    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvktj\" (UniqueName: \"kubernetes.io/projected/2ea2cc76-4433-43d6-ba70-8f753e4a16a8-kube-api-access-wvktj\") pod \"must-gather-8fkp8\" (UID: \"2ea2cc76-4433-43d6-ba70-8f753e4a16a8\") " pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:07.701590 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.701568    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k586c/must-gather-5gktq"
Oct 30 04:34:07.802657 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:07.802620    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/must-gather-8fkp8"
Oct 30 04:34:08.002646 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:08.002619    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k586c/must-gather-5gktq"]
Oct 30 04:34:08.004226 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:34:08.004201    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd042d099_b465_4319_9f62_f32c2267edb0.slice/crio-4f65de60687697df314676b3aaf2343eefdee5165810040cc5b90ade292cc9a4 WatchSource:0}: Error finding container 4f65de60687697df314676b3aaf2343eefdee5165810040cc5b90ade292cc9a4: Status 404 returned error can't find the container with id 4f65de60687697df314676b3aaf2343eefdee5165810040cc5b90ade292cc9a4
Oct 30 04:34:08.006136 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:08.006120    4877 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:34:08.025950 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:08.025920    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k586c/must-gather-5gktq" event={"ID":"d042d099-b465-4319-9f62-f32c2267edb0","Type":"ContainerStarted","Data":"4f65de60687697df314676b3aaf2343eefdee5165810040cc5b90ade292cc9a4"}
Oct 30 04:34:08.097953 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:08.097929    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/must-gather-8fkp8"]
Oct 30 04:34:08.099753 ip-10-0-135-226 kubenswrapper[4877]: W1030 04:34:08.099727    4877 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea2cc76_4433_43d6_ba70_8f753e4a16a8.slice/crio-2cd7e0a4a041c5aef527f73e79f8758d3cabbb2e9fca7d5d8f35829f232feecc WatchSource:0}: Error finding container 2cd7e0a4a041c5aef527f73e79f8758d3cabbb2e9fca7d5d8f35829f232feecc: Status 404 returned error can't find the container with id 2cd7e0a4a041c5aef527f73e79f8758d3cabbb2e9fca7d5d8f35829f232feecc
Oct 30 04:34:09.048126 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:09.048084    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/must-gather-8fkp8" event={"ID":"2ea2cc76-4433-43d6-ba70-8f753e4a16a8","Type":"ContainerStarted","Data":"2cd7e0a4a041c5aef527f73e79f8758d3cabbb2e9fca7d5d8f35829f232feecc"}
Oct 30 04:34:10.061446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:10.060808    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/must-gather-8fkp8" event={"ID":"2ea2cc76-4433-43d6-ba70-8f753e4a16a8","Type":"ContainerStarted","Data":"3d44a2daaf7a1a19b04ef4ba30fe93f90b3b4c78b482ee3df4b2d09fc11b6bbe"}
Oct 30 04:34:10.061446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:10.060867    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/must-gather-8fkp8" event={"ID":"2ea2cc76-4433-43d6-ba70-8f753e4a16a8","Type":"ContainerStarted","Data":"fb337ca992f6993515eaebaa29a15ef15264e310666952d69490f38cc4e67e77"}
Oct 30 04:34:10.098454 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:10.096819    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s9nqc/must-gather-8fkp8" podStartSLOduration=2.282125084 podStartE2EDuration="3.096800021s" totalImagesPullingTime="814.674937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:10.094776847 +0000 UTC m=+5478.523666265" watchObservedRunningTime="2026-10-30 04:34:10.096800021 +0000 UTC m=+5478.525689439"
Oct 30 04:34:13.097823 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:13.097788    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rtd54_bce76cad-b673-431c-b6a4-40298db25172/global-pull-secret-syncer/0.log"
Oct 30 04:34:13.447180 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:13.447090    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gt4dd_af46b57a-5155-4ddd-90d8-bf2eb36f8319/connection-checker/0.log"
Oct 30 04:34:13.598766 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:13.594716    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ptzxj_d1d1d308-d556-4dbc-bd3a-2ec2c838413a/connection-checker/0.log"
Oct 30 04:34:13.745534 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:13.745360    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zwfb4_160c1f29-a378-43d0-97d8-be1cc643e2c2/connection-checker/0.log"
Oct 30 04:34:14.262411 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:14.262376    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sfv5t_940bd241-206c-44c1-9c91-b6eb875bce7c/konnectivity-agent/0.log"
Oct 30 04:34:14.506137 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:14.506106    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-226.ec2.internal_2c7f2093544cf1327cc6ce2b1dd9d362/haproxy/0.log"
Oct 30 04:34:16.354006 ip-10-0-135-226 kubenswrapper[4877]: E1030 04:34:16.353966    4877 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xq4n" containerName="extractor"
Oct 30 04:34:17.748704 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:17.748557    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:34:17.793875 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:17.793827    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:34:22.016446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:22.016191    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:34:22.146446 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:22.145026    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:34:22.989992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:22.989873    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b66c856-5qx55_f34c49c1-2d78-4b79-8505-f68314e4d4b9/authorino/0.log"
Oct 30 04:34:23.245796 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:23.245722    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-npmx5_297ec4ff-cbaf-48ea-a5cd-fab91d07a75e/manager/0.log"
Oct 30 04:34:23.260148 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:23.260100    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k586c/must-gather-5gktq" event={"ID":"d042d099-b465-4319-9f62-f32c2267edb0","Type":"ContainerStarted","Data":"48a66b0c1dfeb6aeae4eb857108aaeec36b9d3df65b3f329edbd68d93978563c"}
Oct 30 04:34:23.260340 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:23.260167    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k586c/must-gather-5gktq" event={"ID":"d042d099-b465-4319-9f62-f32c2267edb0","Type":"ContainerStarted","Data":"d4fc3075a8be5f4c47c8e98dc8bedcbad185c265e7c01b05e57ab3214e520e5f"}
Oct 30 04:34:23.368592 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:23.368560    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gggrh_713ca209-955a-44dd-b634-18e58ef6735b/manager/0.log"
Oct 30 04:34:23.497726 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:23.497639    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8m78v_490cd1fd-4982-4de2-824f-1a3f359dc24d/kuadrant-console-plugin/0.log"
Oct 30 04:34:23.702933 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:23.702893    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8blbd_179cc2a6-a3a9-4f33-a524-95bfabdf7f2d/manager/0.log"
Oct 30 04:34:24.123237 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.123195    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rcpz6_66695834-c9cd-430e-946c-aa57ae794bca/manager/0.log"
Oct 30 04:34:24.530795 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.530460    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k586c/must-gather-5gktq" podStartSLOduration=3.280533984 podStartE2EDuration="17.530439737s" totalImagesPullingTime="14.249905753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:23.298660642 +0000 UTC m=+5491.727550062" watchObservedRunningTime="2026-10-30 04:34:24.530439737 +0000 UTC m=+5492.959329157"
Oct 30 04:34:24.535107 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.535057    4877 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"]
Oct 30 04:34:24.547051 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.546767    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.559926 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.559893    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"]
Oct 30 04:34:24.647597 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.647555    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-sys\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.647809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.647604    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-proc\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.647809 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.647726    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-podres\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.647957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.647841    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645dh\" (UniqueName: \"kubernetes.io/projected/013f77fa-3244-4132-95cf-9bb700173001-kube-api-access-645dh\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.647957 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.647881    4877 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-lib-modules\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749072    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-sys\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749115 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749135    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-proc\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749189    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-podres\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749235    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645dh\" (UniqueName: \"kubernetes.io/projected/013f77fa-3244-4132-95cf-9bb700173001-kube-api-access-645dh\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749259    4877 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-lib-modules\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749534    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-lib-modules\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749594    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-sys\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749627    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-proc\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.749786 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.749664    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/013f77fa-3244-4132-95cf-9bb700173001-podres\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.770853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.770822    4877 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-645dh\" (UniqueName: \"kubernetes.io/projected/013f77fa-3244-4132-95cf-9bb700173001-kube-api-access-645dh\") pod \"perf-node-gather-daemonset-m22pn\" (UID: \"013f77fa-3244-4132-95cf-9bb700173001\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:24.891413 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:24.891053    4877 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:25.246800 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:25.246765    4877 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"]
Oct 30 04:34:25.288722 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:25.288679    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn" event={"ID":"013f77fa-3244-4132-95cf-9bb700173001","Type":"ContainerStarted","Data":"b738354690ab32e489dab56cc192e27a0fa1279c2d46ba6a7b7f39da59680421"}
Oct 30 04:34:25.881408 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:25.881371    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:34:25.979270 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:25.979169    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:34:26.298320 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:26.298285    4877 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn" event={"ID":"013f77fa-3244-4132-95cf-9bb700173001","Type":"ContainerStarted","Data":"2085ba575ee2904eff58e45934400eeb5794d201ebeec1432ec506a35a676ae3"}
Oct 30 04:34:26.298544 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:26.298396    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:26.332564 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:26.332515    4877 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn" podStartSLOduration=2.33250238 podStartE2EDuration="2.33250238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:26.331215796 +0000 UTC m=+5494.760105211" watchObservedRunningTime="2026-10-30 04:34:26.33250238 +0000 UTC m=+5494.761391798"
Oct 30 04:34:27.273772 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:27.273734    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ph5d9_74e50c74-23f8-46b7-98e6-6ac103daba15/cluster-monitoring-operator/0.log"
Oct 30 04:34:27.878853 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:27.878820    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn_1a4cb6aa-0810-497b-a9c8-510595534f09/istio-proxy/0.log"
Oct 30 04:34:27.953374 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:27.953341    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:34:27.998724 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:27.998690    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:34:28.016928 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:28.016901    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g26vv_26cf120d-2eae-4536-9da5-c936e7f9b17a/node-exporter/0.log"
Oct 30 04:34:28.138343 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:28.138248    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g26vv_26cf120d-2eae-4536-9da5-c936e7f9b17a/kube-rbac-proxy/0.log"
Oct 30 04:34:28.260601 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:28.260571    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g26vv_26cf120d-2eae-4536-9da5-c936e7f9b17a/init-textfile/0.log"
Oct 30 04:34:31.411857 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:31.411821    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/0.log"
Oct 30 04:34:31.415178 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:31.415149    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/1.log"
Oct 30 04:34:31.477232 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:31.477180    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fs5rj_11ff9767-6324-495f-be0e-6bca50031925/istio-proxy/0.log"
Oct 30 04:34:31.612442 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:31.612141    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-qzl6k_1057196a-2b55-42a3-a8d4-4918a927d4b7/istio-proxy/0.log"
Oct 30 04:34:31.927693 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:31.927631    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm_cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c/istio-proxy/0.log"
Oct 30 04:34:32.128284 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:32.128235    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ddf86b9f-2wfp9_1569b662-064a-4998-b332-03c6ed48e312/router/0.log"
Oct 30 04:34:32.318002 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:32.317941    4877 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-m22pn"
Oct 30 04:34:32.786992 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:32.786889    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:34:32.871660 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:32.871622    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:34:34.851493 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:34.851457    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-lrntf_ba135d3a-8995-4181-a315-51ae998bba50/networking-console-plugin/0.log"
Oct 30 04:34:36.173747 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:36.173706    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/2.log"
Oct 30 04:34:36.182363 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:36.182336    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qjtcv_3a65983f-48e3-4e8f-a576-2110dfbb2ec2/console-operator/3.log"
Oct 30 04:34:37.576874 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:37.576835    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b66c856-5qx55_f34c49c1-2d78-4b79-8505-f68314e4d4b9/authorino/0.log"
Oct 30 04:34:37.620754 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:37.620719    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-npmx5_297ec4ff-cbaf-48ea-a5cd-fab91d07a75e/manager/0.log"
Oct 30 04:34:37.682854 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:37.682818    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gggrh_713ca209-955a-44dd-b634-18e58ef6735b/manager/0.log"
Oct 30 04:34:37.744849 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:37.744813    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8m78v_490cd1fd-4982-4de2-824f-1a3f359dc24d/kuadrant-console-plugin/0.log"
Oct 30 04:34:37.888448 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:37.888331    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8blbd_179cc2a6-a3a9-4f33-a524-95bfabdf7f2d/manager/0.log"
Oct 30 04:34:38.154945 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:38.154866    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rcpz6_66695834-c9cd-430e-946c-aa57ae794bca/manager/0.log"
Oct 30 04:34:38.261304 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:38.261244    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-prfl7_0201f5d7-a014-4d69-983e-a25e14a48670/manager/0.log"
Oct 30 04:34:38.346516 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:38.346472    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ghfrc_f7a74cb9-1151-42d3-a91c-aafd6cfbc689/proxy/0.log"
Oct 30 04:34:38.876380 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:38.876348    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn_1a4cb6aa-0810-497b-a9c8-510595534f09/istio-proxy/0.log"
Oct 30 04:34:39.018571 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:39.018538    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-2mdnr_b7114a74-db33-4c40-a401-030ffa957f21/volume-data-source-validator/0.log"
Oct 30 04:34:40.186703 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:40.186643    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd7b898fjtrrn_1a4cb6aa-0810-497b-a9c8-510595534f09/istio-proxy/0.log"
Oct 30 04:34:41.578819 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:41.578783    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cfphv_9c1080c7-2fbb-471b-9ea9-b05fc60f95e9/dns/0.log"
Oct 30 04:34:41.697212 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:41.697180    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cfphv_9c1080c7-2fbb-471b-9ea9-b05fc60f95e9/kube-rbac-proxy/0.log"
Oct 30 04:34:42.106245 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:42.106215    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2hl5n_9de66365-30cc-4adc-ad1b-4a6778246af8/dns-node-resolver/0.log"
Oct 30 04:34:42.935244 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:42.935207    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/0.log"
Oct 30 04:34:42.939974 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:42.939950    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/1.log"
Oct 30 04:34:42.979721 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:42.979687    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fs5rj_11ff9767-6324-495f-be0e-6bca50031925/istio-proxy/0.log"
Oct 30 04:34:43.109513 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:43.109478    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-qzl6k_1057196a-2b55-42a3-a8d4-4918a927d4b7/istio-proxy/0.log"
Oct 30 04:34:43.430108 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:43.430075    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm_cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c/istio-proxy/0.log"
Oct 30 04:34:43.626533 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:43.626504    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ddf86b9f-2wfp9_1569b662-064a-4998-b332-03c6ed48e312/router/0.log"
Oct 30 04:34:43.857261 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:43.857225    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5sbs_9da977d0-2885-4889-a18f-9038e7e11272/node-ca/0.log"
Oct 30 04:34:44.257679 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:44.257588    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/0.log"
Oct 30 04:34:44.262680 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:44.262655    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fl76j_d7badd71-7851-4ddb-b54b-86d2707078be/istio-proxy/1.log"
Oct 30 04:34:44.303368 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:44.303333    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-fs5rj_11ff9767-6324-495f-be0e-6bca50031925/istio-proxy/0.log"
Oct 30 04:34:44.433495 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:44.431520    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-qzl6k_1057196a-2b55-42a3-a8d4-4918a927d4b7/istio-proxy/0.log"
Oct 30 04:34:44.744067 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:44.744018    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd8768855-g87vm_cc2f1a6e-d6f6-4a88-bdbb-5104e0317d7c/istio-proxy/0.log"
Oct 30 04:34:44.931825 ip-10-0-135-226 kubenswrapper[4877]: I1030 04:34:44.931792    4877 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ddf86b9f-2wfp9_1569b662-064a-4998-b332-03c6ed48e312/router/0.log"